It's because of the name and history of fighting for women. The movement should have gone for "egalitarianism" if they didn't want people to think of women first/exclusively.
It's why I've vehemently rejected the label of feminist even when I'm in feminist spaces with feminist friends: I'm here for everyone, there's a word for that, use it.