Just curious: do you also lack the ability to visualize things in your mind? For example, I am able to bring up a road map of my city in my mind, figure out the most effective route between two points, and rotate that map in all three dimensions to “look” at it from all angles. My familiarity with the city layout and geography is the determining factor on how easily I can visualize that map. I can also do the same thing with large buildings and their internal layouts.
My wife, on the other hand, has a somewhat similar (but nowhere near as bad) sense of direction as you, and a commensurate inability to visualize objects in her mind. So while she can mentally visualize a soccer ball as a spherical object, she cannot even visualize the hexagonal pattern of pieces, much less (on a traditional soccer ball) how some are white and others black. She doesn’t technically have aphantasia, as she is still able to visualize to a small degree, but I have always suspected her inability to visualize effectively was directly connected to her inability to navigate effectively. She also relies heavily on GPS and maps when navigating anywhere else other than the town she was born in.