The FDA inspector flips to the IT section of your site master file and asks a simple question:

“Show me the last successful restore test for your LIMS.”

Your IT lead produces a tabletop exercise report — a Word document with attendance signatures, a ransomware scenario discussion, and a list of action items from a conference room six months ago. The inspector nods, writes something in her notebook, and moves on. Three weeks later, the 483 arrives:

“Failure to adequately demonstrate that electronic records critical to GxP operations can be recovered from backup. Tabletop discussion records do not provide objective evidence that backup media can be successfully restored without data loss or corruption of audit trails.”

This is one of the top five IT-related 483 findings in the industry. Not because companies lack DR plans — almost everyone has one. But because the plan exists as documentation while the evidence of recovery capability does not.

Three questions define whether your DR program survives an audit: Do you have the right plan? Have you proven it works? And can you prove it works continuously? This post answers all three.

What Auditors Actually Check

When an auditor asks about disaster recovery, they are not really asking for a 100-page policy document. They are testing whether you can answer a chain of questions that maps directly to regulatory requirements:

Auditor Question What They Want to See Regulatory Anchor
What systems are critical? GxP system inventory with criticality tiers GAMP 5, BIA documentation
How quickly must each recover? RTO/RPO per system, justified by business impact NIST 800-34, EU Annex 11 §16
Where are backups? Backup architecture with encryption, immutability, geographic separation 21 CFR Part 11.10(c)
Can you actually restore them? Documented restore test evidence with measured results EU Annex 11 §16, FDA Data Integrity Guidance
Is the recovered data intact? Hash verification, record counts, audit trail continuity ALCOA+, 21 CFR Part 11.10(a)
Who is responsible? Named individuals with backup personnel ICH Q10
What happens to GxP data during an outage? Manual workarounds, disposition procedures 21 CFR Part 211.68(b)
Did you fix what broke? CAPA records linked to test failures ICH Q10, 21 CFR Part 820.100

Miss any link in this chain and you get a finding. The chain is: policy → analysis → plan → evidence of testing → CAPA closure. Most companies have the first three. Very few have clean evidence for the last two.

The Minimum Viable DR Plan

A plan that passes audit is not a thick binder nobody reads. It is a controlled document — versioned, QA-approved, and reviewed annually — that answers every question above with enough specificity that someone unfamiliar with the system could execute recovery. Here is the structure.

Document Control Header

SOP-IT-DR-001. Version-controlled under your QMS. Approvals from IT Head, System Owner, and QA. Periodic review: annual, or after any triggering event (infrastructure change, actual incident, test failure).

System Inventory and Business Impact Analysis

This is the foundation. You must prove you know what is critical. A tiered inventory with documented RTO/RPO per system:

System GxP Classification Tier RTO RPO Backup Method Recovery Site
LIMS Direct GxP 1 4 hours 1 hour Immutable S3 + validated Veeam AWS DR Region
eQMS (Veeva) Direct GxP 1 4 hours 0* Vendor managed + independent export Vendor DR
ERP (SAP) Direct GxP 1 8 hours 4 hours HANA log backup Secondary DC
MES Direct GxP 1 8 hours 1 hour DB cluster replication Warm standby
EMS Direct GxP 1 4 hours 15 min Real-time replication Hot standby
ELN GLP/GCP 2 24 hours 4 hours SaaS + periodic export SaaS DR
AD / DNS Supporting 1 2 hours 0 AD recycle + replication DR DC
File Shares Supporting 2 24 hours 24 hours Daily incremental Cloud restore

*For SaaS GxP systems, RPO 0 means the vendor’s contractual commitment plus your own independent data export. Never rely solely on the vendor.

The tier definitions matter because they drive everything downstream — backup frequency, testing cadence, and how much audit scrutiny you will face:

  • Tier 1 (Critical): Patient safety impact, product release, regulatory submission data. RTO 4–8 hours, RPO ≤ 1 hour.
  • Tier 2 (Important): Supports GxP processes but not directly patient-facing. RTO 24 hours, RPO 4–12 hours.
  • Tier 3 (Standard): Non-GxP, administrative. RTO 48–72 hours, RPO 24 hours.

Risk Assessment

Aligned to ICH Q9 quality risk management. Threats must cover ransomware (the number-one driver of DR activations in 2026), natural disasters, infrastructure failure, vendor/cloud failure, data corruption, and insider threats. For each: Likelihood (1–5) × Impact (1–5) = Risk Score. Document mitigation for anything ≥ 12.

Roles and Responsibilities

Named individuals, not just titles. The DR Coordinator declares disaster and leads recovery. System Owners decide if their system can return to GxP use. QA must approve return to operations — no QA approval means the system stays down. IT Infra and DBA execute the restore. Regulatory Affairs assesses notification obligations.

Every key role needs a documented alternate. If the DR Coordinator is on vacation and the primary DBA is unreachable, the plan must name who steps in.

Backup Architecture

The non-negotiable minimum for GxP:

TIER 1  Real-time replication     Critical systems, synchronous,
                                  RPO ≈ 0, active-passive failover

TIER 2  Daily incremental         All GxP systems, 90-day retention,
                                  AES-256 encrypted at rest

TIER 3  Weekly full               All systems, 1-year retention,
                                  geographically separated copy

TIER 4  Monthly archive           Long-term regulatory retention,
                                  immutable/WORM storage,
                                  quarterly restore verification

3-2-1 RULE: 3 copies, 2 media types, 1 offsite (or air-gapped)

Immutable, air-gapped backup copies are no longer optional. Ransomware specifically targets backup infrastructure. If your backups sit on the same network with the same credentials as production, they are not backups — they are a second copy for the attacker to encrypt.

The FDA’s data integrity guidance defines a backup as a true copy of the original data maintained securely throughout the retention period, including associated metadata. A temporary crash-recovery copy is not a backup. If your backup does not include audit trails, electronic signatures, and associated metadata, it fails the regulatory definition.

Recovery Runbooks

This is where most plans fail audit scrutiny. A runbook that says “IT will restore the system from backup” is worthless. A runbook must be specific enough that someone unfamiliar with the system could execute recovery. The dependency chain matters — restore AD first, then the database, then the application, then interfaces.

A defensible runbook structure:

SYSTEM: LIMS
RUNBOOK ID: DR-RB-001
LAST TESTED: 2026-05-14
RTO: 4 hours   RPO: 1 hour

DEPENDENCY CHAIN:
  Identity Provider → Network → Database → Application
  → Instrument Interfaces → File Storage → Reporting

RECOVERY STEPS:
  1.  Declare incident (timestamp, declarer, reason)
  2.  Notify stakeholders per communication tree
  3.  Freeze affected GxP operations
  4.  Determine last known good state from backup logs
  5.  Provision recovery infrastructure
  6.  Restore database from verified backup
  7.  Restore application layer
  8.  Restore configuration and credentials
  9.  Restore instrument interfaces
  10. Validate connectivity
  11. Verify authentication and authorization
  12. Verify audit trail integrity
  13. Verify representative GxP records
  14. Reconcile data against known baseline
  15. Business owner verifies functionality
  16. QA determines return-to-use status
  17. Resume operations
  18. Close incident
  19. Capture lessons learned

POST-RESTORE VALIDATION:
  □ Record count match (pre-incident vs. restored)
  □ Hash/checksum verification on critical tables
  □ Primary key / foreign key integrity check
  □ Audit trail continuity (no gaps)
  □ Electronic signature verification
  □ Critical report execution test
  □ QA sign-off

Communication Plan

Who can declare a disaster. Call tree with phone (primary), personal mobile (backup), personal email (last resort). Work email and work phone alone are not sufficient — those systems may be the ones that are down. Template notifications for QA, leadership, and if needed, health authorities when supply impact is possible.

Life-Science-Specific Recovery Validation

This is what separates a GxP DR plan from generic IT continuity. A restored GxP system is not automatically “back.” You need a documented process confirming the recovered system matches its last validated state before releasing it to production. This means data integrity verification (ALCOA+ checklist), audit trail continuity check, electronic signature functionality verification, and a formal QA sign-off. Not full revalidation — a lightweight re-qualification protocol. But it must be defined in advance and executed with evidence.

Tabletop Is Not Enough

Here is the uncomfortable truth that the opening scenario illustrates: a tabletop exercise alone will not satisfy an auditor for Tier 1 GxP systems.

Tabletop exercises are valuable. They test whether people know the plan, whether roles are clear, whether communication paths work, and whether the plan has logical gaps. They are low-cost and low-risk. Every DR program should include them.

But they prove exactly one thing: that the team discussed the scenario in a conference room. They do not prove that backups can be restored. They do not prove that RTO/RPO targets are achievable. They do not prove that recovered data maintains ALCOA+ integrity. They do not prove that validated systems function correctly after recovery. They do not prove that interfaces between systems reconnect.

The testing maturity spectrum for DR looks like this:

Level Test Type What It Proves Audit Value
1 Paper review Plan exists, contacts current Minimal
2 Tabletop exercise Roles, decisions, communication Moderate — necessary baseline
3 Component restore test A backup actually restores High — proves recoverability
4 Partial failover Critical system runs in DR environment Very high
5 Full failover + business validation RTO achievable end-to-end, GxP state verified Gold standard

The minimum defensible testing program for Tier 1 GxP systems combines Level 2 and Level 3 at minimum:

Test Type Frequency Systems Evidence Required
Tabletop exercise Semi-annual (annual minimum) All GxP systems Scenario document, attendance, gaps identified, CAPAs
Backup restore test Quarterly Tier 1 critical Restore log, data integrity verification, measured RTO/RPO
Full recovery test Annually Tier 1 critical Complete recovery report with QA sign-off, deviations, CAPA
Communication tree test Annual All DR staff Actual call/message log with timestamps

The Clever Testing Approach

Do not test by destroying production. Test by restoring into isolation:

PRODUCTION ──backup──▶ ISOLATED DR LAB ──▶ Restore ──▶ Test ──▶ Destroy

You can repeatedly test recovery without risking the actual GxP environment. Provision an isolated VPC, restore the backup, run the validation checks, measure RTO, verify data integrity, capture evidence, tear it down. This is the single most valuable thing you can do for audit readiness — and it costs nothing but compute time if you are already in the cloud.

What the Evidence Package Looks Like

Every test must produce a documented package. This is what auditors read line by line:

DR Test #2026-001                                                   
├── Approved Test Protocol (pre-defined success criteria, QA-signed)
├── Test Scenario Description                                       
├── System / Version Under Test                                     
├── Backup Used (date, type, location)                              
├── Recovery Environment Specification                              
├── Expected RTO / Expected RPO                                     
├── Actual RTO / Actual RPO (measured with timestamps)              
├── Execution Evidence (step-by-step logs, screenshots)             
├── Data Integrity Verification                                     
│   ├── Record count reconciliation                                 
│   ├── Hash/checksum validation                                    
│   ├── Primary/foreign key integrity                               
│   └── Audit trail continuity check                                
├── Application Functionality Verification                          
├── Interface / Connectivity Verification                           
├── QA Review and Sign-off                                          
├── Deviations Found                                                
├── CAPA Initiated (with numbers and closure dates)                 
└── Final Test Report

Having failures documented and corrected is actually better evidence of a functioning quality system than claiming every DR test passed perfectly. An auditor who sees two deviations with closed CAPAs trusts your program more than one that claims zero issues ever.

The DR AI Agent: Architecture for Continuous Recovery Assurance

This is where the discipline becomes genuinely interesting as a product problem. The question shifts from “do you have a DR plan” to “can you continuously prove that your regulated systems are actually recoverable?”

The answer requires an agent that does not just generate documents — it monitors, verifies, tests, and produces audit-ready evidence continuously. But in life sciences, the agent cannot operate as an unconstrained black box. It must be built around deterministic tool calling, immutable audit trails, and human-in-the-loop gates to remain validatable under CSA and GAMP 5.

The Architecture

┌────────────────────────────────────────────────────────────────┐
│                      GxP Compliance Boundary                   │
│                                                                │
│  ┌──────────────┐    ┌──────────────┐    ┌─────────────────┐   │ 
│  │ Telemetry &  │───▶│  Diagnostic  │───▶│  Orchestrator   │   │ 
│  │ Signals      │    │  Agent       │    │  (Deterministic │   │ 
│  │ (Monitoring, │    │  (Structured │    │   Tool Calls)   │   │ 
│  │  Backup Logs,│    │   Output)    │    │                 │   │ 
│  │  SIEM)       │    │              │    │                 │   │ 
│  └──────────────┘    └──────────────┘    └────────┬────────┘   │ 
│                                                    │           │
│                                       ┌────────────▼────────┐  │
│                                       │  Human Approval     │  │
│                                       │  Gate (Part 11      │  │
│                                       │  Dual-Signature)    │  │
│                                       └────────────┬────────┘  │
│                                                    │           │
│  ┌──────────────┐    ┌──────────────┐    ┌────────▼────────┐   │ 
│  │ Immutable    │◀───│ Verification │◀───│  Execution      │   │ 
│  │ Audit Log    │    │ Engine       │    │  Engine (IaC)   │   │ 
│  │ (Langfuse /  │    │ (Hash, ALCOA+│    │                 │   │ 
│  │  OpenTel)    │    │  Checks)     │    │                 │   │ 
│  └──────────────┘    └──────────────┘    └─────────────────┘   │ 
└────────────────────────────────────────────────────────────────┘

Six specialized agents, each with a defined role:

1. Diagnostic Agent. Monitors heartbeat failures, snapshot corruptions, backup job failures, and replication latency. Uses structured output (DSPy with Pydantic schemas — IncidentClassification, AffectedAssets, ProposedRunbook) to map live infrastructure alarms directly to approved SOP identifiers. Not creative. Deterministic.

2. Drift Monitor. Daily scan: Are backups running and verifiable? Is the DR site out of sync with production? Are runbooks stale relative to the current architecture? When someone migrates Oracle to AWS RDS, this agent flags that the backup strategy, recovery procedure, RTO/RPO evidence, and DR test all need updating.

3. Runbook Generator. Turns infrastructure-as-code definitions and SOPs into executable, version-controlled runbooks. Not static PDFs — dynamic recovery sequences that adjust based on the actual failure mode. If the primary data center is down and the backup is six hours old, the agent adjusts the recovery sequence and flags the RPO gap to QA.

4. Tabletop Facilitator. Generates realistic scenarios based on your actual tech stack and threat landscape. During exercises, it acts as the inject engine — introducing complications, asking probing questions, taking minutes, and auto-drafting gaps and CAPAs. This has the highest audit ROI of any agent capability because it turns a once-a-year exercise into a repeatable, evidence-producing process.

5. Evidence Collector. During real incidents or DR tests, it collects logs, screenshots, test results, and data integrity verification output. Builds the complete audit package automatically — the DR Test Summary Report with measured RTO/RPO, hash verification results, record count reconciliation, and deviation documentation. Routes to QA for e-signature.

6. Recovery Readiness Scorer. The killer feature. For each GxP system, it maintains a continuously updated readiness assessment:

LIMS — Recovery Readiness: 91%

Backup              ██████████ 100%
Backup Integrity    █████████░  90%
Restore Tested      ██████████ 100%
Dependencies        █████████░  90%
Runbook Currency    ██████████ 100%
RTO Evidence        █████████░  90%
RPO Evidence        ██████████ 100%

Last Full Recovery: 11 months ago ⚠️
Open Findings: 2

Status: AT RISK — Full recovery exercise 11 months old;
        RTO evidence not reconfirmed after CHG-2026-041.

At the enterprise level: 47 GxP systems, 34 recovery-ready, 9 at risk, 4 with unproven recovery. Click any system to see its specific gaps. Click “Generate Test Protocol” to produce a pre-populated test plan. Click “Generate Evidence Package” after the test to produce the audit-ready report.

What the Agent Must Never Do

For GxP systems, the agent’s authority is bounded:

Autonomous (no approval needed): Automatic failover for HA clusters. Backup retry for failed jobs. Scaling redundant capacity. Alerting and notification.

Advisory (recommends to human): DR plan invocation. Vendor escalation. Regulatory notification assessment.

Human-only (agent assists, human decides): Full DR invocation. Communication to regulators (FDA, EMA). Product recall assessment. Production batch disposition. Return-to-operations approval for GxP systems.

Every AI action is logged with user ID, timestamp, and reason — Part 11 compliant. The agent never generates ad-hoc shell commands for infrastructure changes; it calls pre-validated Terraform or OpenTofu modules exclusively.

The Dependency-Aware Advantage

The most valuable thing the agent does is maintain a live dependency graph. When LIMS depends on Oracle DB, Active Directory, storage, instrument interfaces, SMTP, and reporting — and someone migrates Oracle to AWS RDS — the agent detects the change and asks every DR-relevant question simultaneously:

  • Does the backup strategy still work?
  • Does the recovery procedure still work?
  • Has RTO changed? Has RPO changed?
  • Does the runbook need updating?
  • Does the DR test need repeating?
  • Does the validation assessment need updating?
  • Does the SOP need revision?

This is considerably more valuable than a document-generation tool. It is a continuous assurance system.

Build Roadmap

Phase 1 (Months 1–3): Read-Only Advisor. Ingest SOPs and CMDB into RAG. Build the dependency discovery agent. Monitor and alert but take no action. Generate incident assessments and draft runbooks. Value: faster situational awareness, better documentation.

Phase 2 (Months 4–6): Evidence Automation. Build the Evidence Collector that auto-creates DR test reports. Build the Tabletop Facilitator. Build the Runbook Generator for one system (start with LIMS). Value: the biggest audit gap — test evidence — gets automated.

Phase 3 (Months 7–9): Automated Non-GxP Execution. Agent auto-executes non-GxP recovery steps (DNS failover, infrastructure provisioning). Still requires human approval for anything touching GxP data. Value: reduced infrastructure-layer RTO.

Phase 4 (Months 10–12): Full Orchestration. Agent orchestrates end-to-end recovery with human gates. Integrated with QMS for automated CAPA creation. Continuous readiness monitoring. Automated DR test orchestration in isolated environments. Value: audit-ready DR program with minimal manual overhead.

Validation of the Agent Itself

The agent will eventually become GxP-relevant once used in regulated recovery workflows. Treat it as a GAMP 5 Category 4/5 system:

  • IQ: Verify infrastructure, API connections, model version.
  • OQ: Test each capability with known scenarios — simulate a LIMS outage and verify the agent correctly identifies Tier 1 impact.
  • PQ: Run the agent during an actual DR test and compare its performance against the manual process.
  • Periodic review: Re-validate after model updates, infrastructure changes, or regulatory changes.

Version-control prompts, model versions, and retrieval indexes. Run the agent against historical incident scenarios regularly and measure retrieval accuracy and procedural correctness. Auditors will audit the AI agent too.

The Bottom Line

Three things determine whether your DR program survives an audit:

The plan must be specific. Not “IT will restore from backup” but exact commands, exact dependency chains, exact validation criteria, exact RTO/RPO targets per system. Someone who has never seen your LIMS should be able to execute the runbook.

The testing must be technical. Tabletop is the baseline, not the ceiling. For Tier 1 GxP systems, you need documented evidence that a backup was actually restored, data integrity was verified, RTO/RPO were measured, and QA signed off. Do it in isolation — provision an empty environment, restore, verify, tear down. Repeat quarterly.

The assurance must be continuous. A once-a-year test with a pile of evidence artifacts is necessary but fragile. The real product opportunity — and the real audit advantage — is an agent that continuously monitors backup health, flags drift, generates test evidence, and maintains a live readiness score for every GxP system.

The gap between “we have a DR plan” and “we can prove we can recover” is where 483 findings live. Close the gap with real testing first, then automate the evidence with an agent that makes recovery assurance a continuous process rather than an annual scramble.

Research notes: [[Disaster Recovery Plan - GxP Audit Requirements]] · [[DR Testing - Tabletop vs Technical Restore]] · [[DR AI Agent Architecture for Life Sciences]]