Lemmy has a fair number of loud, toxic instances, communities, and users. On the other hand, it's easy to block all of them, and it's practically a requirement to enjoy lemmy.
So block with zero hesitation -- the only people that will give you crap about it are the people that are the problem in the first place.