Imagine calling the difference between people who do stupid things and people who are born with diagnosed mental illnesses "splitting hairs".
It's very, very simple. In one case, you are attacking someone who is completely in control of their mental facilities. In the other, you are attacking people who are literally incapable of defending themselves, from birth. They are not synonymous. If you think that level of punching down is okay, then be as indignant and self-righteous about it as you want, but you deserve to be told.