Does changing default SSH port really make server more secure? Most of scan apps (eg. Nmap) will find it anyway, won't they?

It will find it if it's a targeted attack, but lots of them are just trying on port 22 and moving on it it fails. That said, I don't like changing default ssh port, proper firewall/IDS/IPS is always a better solution.

If it's keypair setup then it doesn't really matter, but once you move away from port 22 at least your logs will look less messy. On port 22 I had at least a login attempt per second, on port 774 I get none except for my own.

It's the equivalent of making sure your administrator username isn't admin.
Goes a long way to confusing opportunists

It stops bots from trying to login. Disabling password login will make it more secure. Changing port just makes your logs tidier.

Fail2ban is your answer... Jail that fucking slutty port so scary gary's cant brutalize her tiny hole...