And that's fair, I guess in that sense it is a true paradox. It just appears a little different in theory and in practice - the theory is the paradox, the practice is not.
Sorry, calling out that it's a social contract is a bit of a knee-jerk response for me, after years of having people whip out the paradox of tolerance as some kind of "gotcha, LIBS!!!" because being tolerant of unfamiliar lifestyles doesn't mean I won't punch a nazi when it's relevant. And that's poorly understood. My rights end where yours begin, and vice versa, but if you start actively infringing on the rights of others and souring that contract, it is our duty as righteous citizens to put you back in your box. Sometimes that means "hey knock it off asshole", sometimes that means hunting down bigots and deleting their kneecaps. Depends what you're guilty of and where.