But this is only the case if you store your passwords in a plaintext file on your phone. Something that I hope nobody would be dumb enough to do, but I guess many people would.
If you have an encrypted password manager like Bitwarden or so where you have a single long password to open and get at your other long secure passwords, then it is essentially a different factor than your phone, right? Since having the phone unlocked would do nothing to help the attacker get to your password vault.