Disaster Recovery Plan

Complete loss of primary AWS region. RTO target: 4 hours (Tier 1: 1 hour). RPO: 1 hour (Tier 1: 15 minutes).

1. 1
   Detection T+0 — Automated

   CloudWatch Synthetic Monitoring detects failed health checks across all primary region endpoints. GuardDuty and Route 53 health checks confirm. PagerDuty pages on-call Infrastructure Engineer immediately. AWS Service Health Dashboard checked for regional incident declaration.
2. 2
   Assessment T+10 min

   On-call engineer confirms: regional failure vs. application-level failure vs. network issue. Attempts to reach primary region control plane via alternative network path. Checks AWS Personal Health Dashboard for confirmed regional event. Documents assessment outcome in #dr-incidents Slack channel.
3. 3
   DR Decision T+15 min

   Incident Commander (CTO) makes GO/NO-GO decision to invoke DR failover. If primary region unavailable for >15 minutes with no AWS ETA: GO decision. If AWS ETA <1 hour: monitor and reassess. GO decision triggers immediate notification to entire DR team.
4. 4
   Failover Execution T+15 to T+75 min

   Infrastructure Lead executes DR runbook: (1) Promote RDS read replica in DR region to standalone primary. (2) Update application configuration in DR region to point to new DB endpoint. (3) Scale up ECS tasks in DR region (Terraform apply from pre-prepared DR config). (4) Update Route 53 DNS records to DR region load balancer (TTL pre-set to 60 seconds for fast propagation). (5) Verify application health via DR region endpoints — all critical paths tested. (6) Confirm rate engine is processing correctly.
5. 5
   Client Communication T+60 min

   CEO approves P1 communication template. Status page (status.propeter.com) updated with incident status, affected regions, and ETA. Proactive email sent to all affected clients via SendGrid (or Twilio SMS if SendGrid affected). Enterprise clients receive direct call from Customer Success Manager.
6. 6
   DR Operation & Primary Restoration T+4 hrs onward

   Platform operates on DR region. Propeter monitors primary region recovery. When primary region available: run integrity checks on primary DB, compare with DR region DB (row counts, audit trail). Plan cutback to primary during next maintenance window to minimise client impact. Cutback is transparent to hotel clients.
7. 7
   Post-Incident Review Within 5 business days

   Blameless post-mortem conducted by CTO, CISO, and Infrastructure Lead. Timeline, contributing factors, and impact quantified. Action items assigned with owners and deadlines. Post-mortem report shared with affected Enterprise clients on request.

Accidental or malicious corruption/deletion of data in PostgreSQL. PITR is the primary recovery mechanism.

1. 1
   Detection T+0

   Automated data integrity checks run every 15 minutes and compare row counts against rolling baseline. Anomalous change (>10% row count drop in any table) triggers immediate PagerDuty P1 alert. Alternatively: client or internal report of data inconsistency.
2. 2
   Immediate Containment T+5 min

   Put database into read-only mode immediately to prevent further writes. If corruption appears write-path related: disable application write endpoints via feature flag. Snapshot current (corrupted) state for forensic analysis — do not overwrite. Revoke all application database credentials; rotate immediately to prevent further corruption if credentials are compromised.
3. 3
   Root Cause Assessment T+15 min

   CISO and Infrastructure Lead assess: accidental operational error, application bug, or malicious act. If malicious: invoke full incident response (Section 6 of Security Overview). If operational error: proceed directly to PITR. Identify the exact timestamp of corruption from CloudWatch Logs and RDS audit trail.
4. 4
   Point-in-Time Recovery (PITR) T+30 to T+90 min

   Initiate RDS PITR restore to 1 second before identified corruption timestamp. PITR restores to a new RDS instance (original instance preserved for forensics). Validate restored DB: run data integrity suite, spot-check specific records, confirm audit trail continuity. Identify and quantify data loss window (difference between PITR timestamp and corruption time).
5. 5
   Application Cutover to Restored DB T+90 min to T+3 hrs

   Update application DB connection string to point to restored instance. Gradually re-enable write endpoints. Monitor application error rate for 30 minutes before declaring recovery complete. Issue new application DB credentials scoped to restored instance.
6. 6
   Client Impact Assessment and Notification T+2 hrs

   Identify which tenants had data in the corruption window. Quantify: which bookings, rate changes, or forecasts may need to be re-entered. Notify affected hotel clients directly with specific details of the data loss window (if any). Assist clients in reconciling their PMS data against Propeter if needed.
7. 7
   Post-Incident Review Within 5 business days

   Full root cause analysis. Bug fix deployed and tested before any write traffic resumes in production. Post-mortem distributed to affected clients. Preventive controls evaluated and implemented.

Volumetric or application-layer DDoS. Primary mitigation is automated; DR failover is last resort.

1. 1
   Automated Mitigation

   AWS Shield Standard automatically detects and mitigates volumetric network-layer DDoS. WAF rate-limiting blocks high-volume IP sources. CloudFront absorbs traffic — edge network scrubs malicious requests before reaching origin.
2. 2
   Manual Assessment

   If automated mitigation insufficient: Infrastructure Lead analyses WAF logs to identify attack pattern. Applies targeted IP or geo-based block rules. Engages AWS Shield Response Team (SRT) if available (requires AWS Shield Advanced — on roadmap).
3. 3
   Rate Engine Priority Mode

   Implement read-only mode for non-critical dashboard features to reduce origin load. Rate engine API given priority compute allocation. Cache TTLs extended to reduce origin hits during attack period.
4. 4
   Regional Failover (Last Resort)

   If primary region completely saturated: Route 53 DNS failover to DR region. DR region serves traffic while origin region is protected. Client notifications sent via email (not affected by web DDoS).

Critical sub-processor unavailable. Each has a defined fallback mode.