Not a fan of them, but I am inclined to agree with you on this one. Appearance of corruption for sitting out and letting the decision default, or much stronger appearance of corruption by sitting for it.
Unless sitting for arguments and abstaining from submitting or signing on to an opinion is an option… but even then, sitting for the arguments of a case you’re a defendant of is a bad, bad look, comes with no guarantee that you actually will abstain from opinion, and might not even be a valid way around quorum rules.