That's the problem I have when people of that slur use it. And worse, they act like it's not a big deal. There's offensive words I can use because of my skin tone that would absolutely get any non-colored person choked out.
But you nailed it. If I brush it off like it doesnt offend/isn't a disgusting word, then I am giving permission to others that it's okay to say.