A regulated digital-asset fintech · March 2026

Surviving the loss of an AWS region

Drone strikes took down AWS me-central-1 — reported as the first confirmed military attack on hyperscale cloud infrastructure. Databases and backup snapshots were rendered unrecoverable.

The recovery path was an Airbyte → MWAA → Redshift data pipeline I had provisioned months earlier — never designed as a DR mechanism.
Owned the DevOps side of a 2-week migration to AWS Jakarta: environments, Secrets Manager, SSM Parameter Store, deployment validation, cross-team coordination.
Snapshot-based DR failed for real. We rebuilt DR strategy around continuous replication paths we can prove work.

2 weeks

Region-to-region recovery, end to end

100%

Of databases and snapshots unrecoverable in-region

Data pipeline that became the accidental lifeline

severitySEV-1 — total loss of the production region

triggerDrone strikes destroyed AWS me-central-1 (UAE) infrastructure — reported as the first confirmed military attack on hyperscale cloud

impactAll in-region databases and backup snapshots unrecoverable; some customer data corrupted and manually restored

root causeDR design assumed snapshots survive a disaster. Physical destruction of the region proved the assumption wrong

recovery~2 weeks to full production in a new region, via a data pipeline never designed for DR

timeline

T+0Strikes hit the region. Every database and backup snapshot in me-central-1 is rendered unrecoverable — not partially, gone.
T+daysThe mysqldump backups stop working. The intended DR path is officially dead.
week 1The Airbyte → MWAA → Redshift analytics pipeline is identified as the only surviving copy of production data. Export from Redshift, import into AWS Jakarta begins.
weeks 1–2DevOps line held: environment configuration, Secrets Manager, SSM Parameter Store, deployment validation, cross-team coordination. Corrupted customer data manually verified and fixed with backend and QA.
T+2 weeksProduction running from ap-southeast-3. DR strategy rebuilt around continuously verified replication paths.

What actually saved the company

The data team had an EC2-hosted Airbyte instance and AWS MWAA (managed Airflow) streaming data into Redshift — a pipeline I had provisioned and set up for them earlier. It was never designed as a DR mechanism, but it became the recovery path: data was exported from Redshift and imported into the AWS Jakarta region.

Honest framing

I was not the person who “led the rebuild” or achieved “zero data loss.” Some customer data was corrupted and required manual verification and fixes alongside other teams. Backend and QA did the heavy lifting on feature correctness. I made the earlier infrastructure decision that turned out to be the lifeline, and I held the DevOps line for two weeks.

The lesson

Snapshot-based DR is fragile in a real disaster. The pipeline saved us almost by accident. Afterwards, we rebuilt the DR strategy around continuous replication paths we know work — not runbooks we hope work.

AWSAirbyteMWAARedshiftSecrets ManagerSSM