Our disaster recovery plan exists on paper, but it has never been tested

The situation

There is a disaster recovery document. It was written for an audit, a regulator or a customer, and it lists recovery time and recovery point objectives for each system. There is a secondary site or a second cloud region. Backups run every night.

Nobody has ever tried to recover the whole business from it. When you ask how long a real recovery would take, the answers range from a few hours to “we would have to see”. You suspect the honest answer is that nobody knows.

Why this is hard

• DR plans age faster than systems change. Every new application, integration or cloud service added since the plan was written is probably missing from it.
• Backups are tested as files, not as a business. Restoring a database is not the same as bringing up the application, its identity, its DNS, its integrations and its users.
• Testing feels dangerous. Teams fear that a DR test will itself cause an outage, so it is postponed.
• Cloud changes the DR model. Multi-region designs, managed services and infrastructure as code make some recovery far easier and other parts, such as identity and data consistency, more subtle.
• Regulators and large customers increasingly ask for evidence of testing, not for the plan.

What goes wrong when it is ignored

• A real incident reveals that recovery takes days, with the business losses that the recovery objectives were supposed to prevent.
• The organization discovers during an outage that the backups are incomplete, encrypted with lost keys or restored to a place where the application cannot run.
• An audit or regulatory review rates the DR capability as unproven, which affects certification or accreditation.
• Management believes the organization is protected and makes decisions on that basis.

How I approach it

1. Compare the plan to the estate. I inventory the systems that actually matter to the business today and check each one against the DR document: is it covered, are the objectives realistic, does the recovery procedure exist?
2. Review the architecture behind the recovery. Backup design, replication, secondary site or region, identity, DNS, network and the dependencies between systems. Many plans fail at the dependencies rather than at the data.
3. Fix the recovery design before testing. Where the architecture cannot meet the stated objectives, I propose changes: different backup tiers, cross-region replication, infrastructure as code for rebuilds, documented identity and DNS failover.
4. Design a safe test program. Starting with tabletop exercises and single-system restores, moving to isolated recovery of a group of systems and, when the organization is ready, a controlled failover. Each test has a scope, a success measure and a rollback.
5. Run the test and record what happened. Actual recovery times, failures, missing steps and surprises are documented without blame.
6. Update the plan into something living. Runbooks per system, an owner per runbook, a test calendar and a change process so new systems are added to the plan when they go live.

This applies equally to AWS, Google Cloud, OCI, stc cloud and Zain Cloud estates, to on-premises environments and to the hybrid arrangements most Saudi organizations actually run.

What you receive

• A gap analysis between the DR plan and the current estate.
• An architecture review of the backup, replication and failover design, with recommended changes.
• A DR test program with scoped exercises, success criteria and safety measures.
• Test reports with measured recovery results and corrective actions.
• Updated recovery runbooks and a maintenance and testing calendar.

Where to start

Bring the current DR document and a list of your most important systems to a free 30-minute Architecture Decision Review at /free-architecture-review/. I will tell you where the plan is likely to fail and what the first safe test should be. The contact form is the alternative.