But the fediverse isn't them taking that data, it's you giving your data to them. It's you placing your data directly on their server.
It's more than my flyer metaphor, it's you literally placing your flyers in someone's house and then saying "but you can't do x y or z". Even if morally you are in the right, how would you ever enforce that or prove something in court? You still have the hurdle of "if you didn't want them to have it, you shouldn't have handed it to them"