Why is it hard to improve privacy on IM?

I am improving again my online privacy these days. I was not using Twitter anymore for a few years, so I created a Mastodon account these days. I am still find interesting people over there, but it was quite simple.

But for IM, things get harder. What to use? XMPP? Matrix? Both appears to be good and bad sides. And it makes no sense I just install any of them and try to connect to random users. I actually need to make people I know to use the software as well.

Not only that, I will still need to keep other APPs on my phone, as they are necessary for some people. So I will still need a bunch of installed software.