We have to differentiate between physical and cybersecurity.
Are you more likely to physically lose your smartphone you carry around with you all day than your full ATX desktop standing in your office? Yeah.
But let's consider the consequences for a moment.
If someone physically stole your desktop, chances are that at least a part of your data isn't encrypted, the boot sequence probably isn't (at least completely) verified, and your OS is wide open.
There is little to no real isolation in most desktop setups. Once somebody managed to gain access to your system, it is outright trivial to steal your browser sessions, modify commands or run some code, at least in userland.
Physically stealing your smartphone is easy. But a modern smartphone is usually protected by verified boot and a password+fingerprint/Face ID combo. Unless you take active steps to compromise the security of the phone like rooting/jailbreaking it, disabling verified boot or disabling the passcode, it's pretty hard if not near impossible to gain access to your data or modify it in a harmful way. If you visit an infected site or install an infected app, the damage is usually confined to that app's data and the data accessible to it by permissions you probably had to allow to be set in the first place.
Now that's speaking to your usual bad actors and usual setups. Exceptions, as always, make the rule. As soon as a sufficiently motivated and technically able actor with access to 0-day exploits, like a state actor, targets you for some reason, all bets are off. But even in this case, due to the advanced verified boot chain on most modern smartphones, those exploits rarely have the ability to survive beyond a reboot.