To add to what everyone else is saying, it's often self inflicted via culture. When you're brought up to treat people who look different as not part of your group, that's hard to unlearn as adults and so it just self perpetuates and you end up with groups of people that never interact, sometimes fearing anyone not in "their group" for no other reason. Or even for flawed inaccurate reasons.