This may take us down a bit of a rabbit hole but, generally speaking, it comes down to how you route traffic.
My firewall has an always-on VPN connected to Mullvad. When certain servers (that I specify) connect to the outside, I use routing rules to ensure those connections go via the VPN tunnel. Those routes are only for connectivity to outside (non-LAN) addresses.
At the same time, I host a server inside that accepts incoming Wireguard client VPN connections. Once I'm connected (with my phone) to that server, my phone appears as an internal client. So the routing rules for Mullvad don't apply - the servers are simply responding back to a LAN address.
I hope that explains it a bit better - I'm not aware of your level of networking knowledge, so I'm trying not to over-complicate just yet.