For me it is usually due to how incessantly preachy and judgemental some vegans are. I respect their choices and consistent choice of morality. But people tend to get annoyed when someone else feels the right to dictate their morality to them. See also: religious nuts.