"Our disaster recovery plan goes something like this: Help!… Help!" — Dilbert (Scott Adams)
In the early 2000s, a virus found its way onto the network of a SaaS company I was working with. In an estimated ninety minutes, the virus had erased or corrupted nearly 40% of our source code and over 30% of our archived data.
Restoring the archived data from backups was manageable. The source code was a different problem. The virus had compromised both our production server and our test server — which had been serving as the production backup. Both machines were on the same network.
What saved us was that developers around the world had various segments of the source code saved on their local machines from prior work sessions. We spent over a week piecing the codebase back together from those fragments. The service was partially restored after a week. Full functionality returned in another two weeks. Complete restoration of data mining capabilities took over two months.
The lessons were direct: more backups, stored in more locations, with no two storage systems on the same network or under the same logical control. And any backup plan that has never been tested is, by definition, untested.
Roadmap Considerations
◆ Define a disaster recovery strategy that explicitly addresses both recovery time objective (RTO) and recovery point objective (RPO). ◆ Assess whether your business model can withstand the slower recovery of a cold backup — or whether you need hot standby infrastructure. ◆ Quantify your cost of downtime. That number should drive your redundancy investment, not the other way around.