What Hardware specs should I focus on when buying a server which runs virtual machines using ProxMox or XenServer?

Quantity, stability, and performance of memory and storage.

Depends on what you'll be doing on those VMs

Depends on what kind of workload the VMs will be doing, but you can never have enough RAM. Storage is also important - if the VMs are going to be running high IOPS applications I would consider SSD or NVME backed storage.

@RememberMe @vektor Mainly services like Nextcloud, Emby (mediaserver), Git. Currently its all running on a small HP Micro Gen8. Maintaining that block of shit is not that easy. And I want to try new apps + development from time to time without having the risk of fucking up my whole system.