9

Worst prod scenario experienced - on site in small African country working on CRM/billing system my colleague was testing some new SQL and after finishing decided to drop and recreate the DB. She thinks the process is very slow and suddenly realizes she is dropping the prod DB. In a panic she shuts down the system and starts doing a restore from tape, but is so stressed out she writes "tar cv" instead of "tar xv" and overwrites the backup with the broken DB. Took a while to clean that one up...

Comments
  • 1
    Wow that is a nasty perfect storm. Did you then use the previous day's incremental / full tapes to restore?
  • 0
    @Jumpshot44 Ended up with a lot of manual re-entry + figuring out account balances by reprocessing call records, but in the end the base of it was the previous backup.
Add Comment