I don't believe this to be true. Fairness only matters to people who value fairness. Many people value fairness, but it is irrational to believe that everyone values fairness. Some, not most or even many, don't care about fairness fundamentally. For these people, interesting fairness does nothing for them. These are the people we need to protect others from while also providing an environment that didn't necessarily mean removing or killing them.