I don't buy the whole "the more users a software has, the better it gets" rhetoric. Historically this has been the opposite of the case. There's an even higher users-to-contributors ratio amongst the general population. Not all users share the same respect for the philosophy behind FOSS.
If the driving force behind design decisions becomes "what keeps people happy so they'll keep using our software" and not "freedom," there's now a practical incentive to sell out and introduce more Intellectual Property shenanigans into the ecosystem. After all, it's a lot easier to hire devs and churn out new features and keep the software actively developed for the foreseeable future if there's money in it. And the only way there can be money in it is if there are proprietary licenses shitting up the place, and Shit As A Service suscription models as far as the eye can see.
Linux always has been, and should always continue to be, about freedom. If that freedom comes with user-friendliness, great! If not, then we have to pay the price: taking responsibility for the tools and tech we use and learning how to use them properly and contributing to them to maintain a community of likeminded people. Otherwise, we're not worthy of the freedom and the responsibilities it entails.
I get your point about elitism and gatekeeping. We're no better than Windows users or Mac users or any other OS' users. We just have a set of values unique to our community, and they have sets of values that differ. We also shouldn't be throwing users under the bus in the name of politics, but part of what makes Linux slightly more bearable is the way the driving philosophy of Free Software is evident throughout. Linux is better than it could be because it attracts the people who want to be here for the community's values, not the people who have to be coaxed and coerced into accepting the values to use the "best"/"easiest"/"friendliest" software.