This was not about being scared and more of a social contract
They would not select judges on the end of a presidents turn, based on the trust that the other party wouldn't either
As soon as one party stops holding to this unwritten agreement (as Republicans did under Trump), there is little to no incentive for the other party to continue abiding to it