As a non-American who didn't grow up with imperial, I still prefer it for fantasy. Metric sounds too modern and scientific. Also I feel like I have more room to fudge distances because it already sounds imprecise.
Would be a fun bit to make the players uses metric in a magitech world though.