I completely agree. Gendered pronouns are not helpful and at this point only confuse things. I'm just glad English doesn't have gendered nouns, too, like Latin-based languages.
Anyway, the fact is that they/them has become "gendered" in the sense that it's now a preferred pronoun for a lot of people, mostly androgynous enbies, so its implicit meaning has changed. Sure, it's still used as a non-gendered pronoun for hypothetical people, but when used for a real, known person, it has the same implication as he/him or she/her - that they appear to be a certain gender, enby in this case.
I'm a clearly masculine person - I've got a beard and I wear masculine clothes. I personally wouldn't be offended, but I would think it very odd if someone saw me and thought they/them was an appropriate pronoun for me. If masculinity was as important to me as it is to most men, I could see myself getting offended at someone implying that I appear androgynous. Same as if an enby was referred to as he/him or she/her. Cisfolk's emotions are just as valid as valid as enbies'.