There is this thing that exists called a museum. Some shit belongs in a museum, some doesn't, and some doesn't belong in the public sphere.
I personally haven't yet heard the argument against Penn, but by and large the whole scare over destroying history by removing things like nazi/confederate statues is overblown at best, and racist at worst.