Digital audio out cable to RCA
The audio out is an analog signal meaning you could directly plug headphones or speakers to that port and get sound, because it is outputting an analog signal. But that sound will probably be at a low level typically -10dbu and will require amplification, so it wouldn't be loud enough for headphones you would need speakers with amplification.
The optical out is a digital signal meaning if you would listen to that signal on its own it would sound like random digital noise beeps clicks and what not. So you have to plug the optical cable into a device that converts digtal data into analog sound. Most home theatre sound systems can handle this.
Quality speaking you would want to use optical because you can buy a device that can convert the digital data into a higher quality analog sound.
Where as the audio out will be converting the digital data into analog sound via the internal digital to analog converter(DAC), built into your tv. And even if you have a 5, 000 dollar expensive 4k tv it will still have the same crappy DAC that some old low level 2005 plasma tv. TV companies spend the bare minimum on DACs.