Optical audio Digital cable to 3.5mm jack
Can I connect a 5.1 speaker with a 3.5 mm jack to my TV (the TV has 3.5 mm output jack)? If yes will it be as good as a 5.1 home theatre system?
The first part is yes, you can connect a 5.1 surround sound system to an analogue stereo headphone output (which is what your 3.5mm stereo jack socket is).
The second part is tricky, as it depends on what the TV manufacturer has decided to to with the MPEG2 or Dolby Digital signal riding alongside the picture of a broadcast.
The TV manufacturer can choose to present a simple down-mix of the 5.1 information, as a stereo, 2.0 signal. In which case, as others have stated, you would not fully benefit from any subsequent decoding, as the extra centre and stereo surround channels would be lost. At best, a Dolby ProLogic II setting would be somewhat able to re-create a Centre and Left Surround / Right Surround, as well as a Sub set of signals, using phase relationship information and its crossover filter network, but the result would not be entirely faithful to the panning done by the re-recording mixers in the auditorium.
Even worse: the TV manufacturer could choose to send down that headphone output, only the Left and Right legs of the 5.1 surround signal! This is a terribly bad solution, as the dialogue is seldom panned to anything else than the Centre channel, and would not be present at all on the 3.5mm jack headphone socket. The Dolby ProLogic II algorithm would be able to somewhat emulate the missing channels, but would definitely not be able to re-create the missing dialogue out of thin air!
The TV manufacturer thought that someone might connect their 5.1 surround sound system via this connector (a far stretch already), and therefore, included in their down-mix process, the necessary Dolby Surround processing to have the resulting 2.0 signal be a LT-RT (Left Total - Right Total) format analogue signal. In which case, a proper Dolby Prologic II decoder can use phase relationship and frequency content, to recreate the original 5.1 channels - although this is in no way as precise as connecting the TV to the surround sound system via the dedicated Digital Output(s)
This can be a single RCA connector
a TOSLink optical
or you could also use audio-enabled HDMI (not all HDMI protocols carry digital audio alongside picture information)
All of the above applies to the media player's built-in 3.5mm headphone socket, if it has one.