This is actually how I do things when working on remote machines. I have far too many monitors, so dedicating on of them to a handful of btop/nvtop terminals works pretty well.
I admit that it's a less than perfect setup though, and a single program which could handle the remote connections internally and display an aggregate would be nice.