Your story basically mimics mine. Guitar and drums, always wanted to play with others but at 50 years old, I jam every day by myself at home.
Other than about 6 months in my 20s, I've not played music with others. I want to but I don't interact with others and the thought of trying to is overwhelming.
I hope you overcome your aversion. The 6 months I did jam with others was magical.