Rant at myself for being a moron.

Manually tried moving some InnoDB stuff around on a local server and corrupted a load of databases, with no backups since June 28th. Spent all day yesterday trying to recover with no luck 😑

Lesson learned: need to backup databases more regularly. Any good tools?

Best backup tool should be what the database provider uses. A database is a raw file system, rather than copy the entire file system it is more efficient to just backup the data within it.

for mysql/mariadb i use a shell script and cron. simple and works great.

postgresql well I'm gonna take that one soon ;)

the dump files are then backed up from my backuppc server

Lack of proper backup is prevalent in lots of companies so you are not alone. It is critical to get it right. Also, to do it right system RTO and RPO must align with the most aggressive business RTO and RPO needs. In my experience I have found often that is not the case. Unfortunately, it is discovered during a crisis when the business expectations are not met.

Your company should do a BIA and an AIA to determine if there are business to system gaps. Also, and I know it can be complex in larger orgs, go upstream and downstream to insure you do not have gaps due to system-system dependencies. Hope that helps. If you need more info please let me know. By the way, 72% of companies that incur a crisis and do not have a tested BC and DR plan go out of business within 3 years. So management should take it seriously if you request funding.