If a black person called me a cracker it wouldn't scare me very much because I know that I have the privilege of being white, and I know that, whether I like it or not, the entire white-supremacist apparatus of the state is likely to prevent me from having to deal with any racial oppression. Those individual people could still hurt me, but so could anyone. They are still far less likely to attack me because they know how dangerous that would be for them, but a white person attacking a black person is more likely to be protected.
When you get called "cracker", you know that you don't have that apparatus behind you. You know that cops, the legal system, and myriad other racist things that you probably know better than me, are still going to treat you as non-white. So the "cracker" attack hurts because you're being excluded from the community that would otherwise protect you when the apparatus of the state won't. I'm not trying to whitesplain this to you, that's very much what you described when you called it "rage and absolute isolation".
That's why the attack hurts you as a mixed-race person, but not me as a white person. I do not experience that isolation. That's the difference.