As I get older and the abuse I put my ears through starts showing up, I completely agree. After upgrading my music library to FLAC from VBR mp3s, I stopped having the, "Oh! There's a subtle instrument going on in this part of the song!" moments.
It doesn't stop me from trying to listen to the highest quality music formats that I can get my hands on, but I 100% know if I think there's a difference to my mid-40s ears, it's probably a placebo.