Mike Has MS
Recovery Test
My company has a policy that we do a full-scale recovery test of the database on our ERP system twice a year. That test was last week. Being a somewhat competent DBA, I should be able do this with half my brain tied behind my back. It wasn’t as easy as it should have been.
Keep in mind that I restore these databases fairly regularly, so the test should have been a formality. But this time, I had to have Derrick, our network guy, get the backup file from the long-term storage that comes back from the mine instead of my usual restores from local storage. I sent Derrick the name of the file that I needed to recover. I didn’t even care of the date on the file to satisfy this requirement.
The restore finished really quickly—almost too quickly. When I restored the database, I went to pull data from a table that would prove we had successfully restored the database. I was horrified when SQL informed me that the table I wanted didn’t exist. I scratched my brain, trying to figure out what was going on. I restored the file again with the same results. I watched my life flash before my eyes. How the ehell was I going to tell my boss that I wasn’t able to restore our ERP system from a month ago? I went back and realized that Derrick had restored the wrong backup file. After restoring the right file, my database restore ran better and the table existed again. Lesson learned: Because this system has multiple databases, the backup files can be easily mixed up. Double-check the filename before panicking.
Next came a test for our corporate controller to sign off on. I would grab a transaction from the AP invoices table so he could look it up in the system to validate that the data was entered on that date. Keep in mind that the backup was taken on Sunday, May 3. That means I should have data from Friday, May 1. As I’m looking at the data, I see transactions from Thursday, April 30. Why could I not see data from Friday? Again, I start scratching my head, trying to figure out this out. I was completely baffled by this one. I walked up to Finance to talk to Cindy in Accounts Payable. Would you believe the entire Finance team took that Friday off to go on a canoe trip?
Problem solved. After a few setbacks, my DR test passed. Now I have to document everything and get the controller to sign off. That’s usually the hard part. Not this time.
Next entry: Let the River Run
Previous entry: Ready for the Eighties