If we were raised in a society where everyone was covered head to toe in figure-obscuring clothes, we'd probably be equally blown away by midriff or tight/short pants. And if we were raised in a nudist society, none of it would phase us.
From where I'm standing, it seems like a question of how much we want to desensitize ourselves to. There's certainly such a thing as too much and too little, so surely most people could agree that the line shouldn't be drawn at a nudist society, right? For me, boobs elicit the same level of response from my body as butts and genitals so the idea of boobs not being behind that line feels disconcerting.