Cloud Computing
High Availability and Disaster Recovery: Availability Maths, RTO, RPO and DR Patterns
Advanced11 min readUpdated
High availability keeps a system serving through component failures; disaster recovery brings it back after something larger. This guide covers the arithmetic behind availability targets, how to derive RTO and RPO from business impact, the redundancy and failover patterns that implement them, the four standard DR strategies and their costs, and how to prove any of it works with game days.
What you will learn
- Calculate composite availability across serial and redundant components and read an SLA critically
- Identify failure domains and the correlated failures that defeat naive redundancy
- Derive RTO and RPO from business impact and translate them into an architecture
- Compare backup-and-restore, pilot light, warm standby and active-active on cost and recovery time
- Design database failover and understand why it dominates the recovery timeline
- Run a game day that produces evidence rather than reassurance
Two disciplines are routinely spoken about as one. High availability is about surviving the failures you expect — a machine dies, a disk fills, a data centre loses power — without a human being involved and without users noticing much. Disaster recovery is about the events you cannot absorb in place: a region-wide outage, a corrupted database replicated everywhere, an account compromised and its resources deleted. They use different mechanisms, cost different amounts, and fail in different ways, and a system can be excellent at one and useless at the other.
This guide starts with the arithmetic, because availability targets chosen without it are wishes. It then works through failure domains, the redundancy patterns that raise availability, the recovery objectives that define disaster recovery, the four standard DR strategies and their economics, and finally the testing without which none of it can be claimed.
Availability arithmetic
Availability is uptime divided by total time, conventionally written in nines. What matters is the allowance it implies.
| Target | Downtime per year | Per month | Realistic without automation? |
|---|---|---|---|
| 99% | 3.65 days | 7.3 hours | Yes |
| 99.9% | 8.77 hours | 43.8 minutes | Yes, with on-call |
| 99.95% | 4.38 hours | 21.9 minutes | Needs redundancy |
| 99.99% | 52.6 minutes | 4.4 minutes | No — recovery must be automatic |
| 99.999% | 5.26 minutes | 26 seconds | Only for narrowly scoped services |
Four nines allows less than five minutes of downtime a month — shorter than the time it takes to wake someone and have them open a laptop. Anything at or above that level must recover without human intervention, which is an architectural statement, not an operational one.
Components in series — every one required for a request to succeed — multiply. A load balancer, an application tier, a cache and a database, each independently at 99.99%, give 0.9999⁴ ≈ 99.96%, roughly three and a half hours a year, before counting your own bugs and deployments. Every dependency you add lowers the ceiling.
Redundant components in parallel multiply their failure probabilities instead. Two instances each with 99% availability fail together only 1% × 1% of the time, giving 99.99% — provided the failures are independent. That proviso carries the whole argument, and it is usually false.
Failure domains and correlated failure
A failure domain is a set of resources that can fail together. Naming them is the core skill, because redundancy only helps when the copies are in different domains.
The physical hierarchy runs: process, host, rack, data centre, availability zone, region. Providers expose the zone as the unit you place against, and spreading instances across zones is the standard defence against a data-centre event.
The domains that actually cause incidents, however, are rarely physical:
- Shared dependencies. Three instances in three zones behind one load balancer, all reading one database in one zone. The database is the domain.
- The control plane. Provisioning APIs, autoscaling and IAM are regional services. During a large regional event, the very APIs you need to launch replacement capacity are the ones degraded. Recovery paths that require creating new resources are weaker than paths that only require shifting traffic to resources that already exist — this is called being statically stable.
- Configuration and deployment. The most common cause of correlated failure is a change applied everywhere at once: a bad config push, a schema migration, an expired certificate deployed by the same automation to every replica. Redundancy offers no protection here; staged rollout does.
- Capacity and quotas. A failover that doubles load in the surviving zone succeeds only if that zone had spare capacity. Scaling to two zones and sizing each for exactly half the traffic guarantees a brownout when one fails.
- Shared fate in software. Every instance running the same code hits the same memory leak at the same uptime, and the same poison message crashes every consumer.
Two patterns limit correlated damage. Cell-based architecture partitions users into independent cells, each with a full stack, so a bad cell affects a known fraction of customers. Shuffle sharding gives each customer a random subset of workers, so one abusive tenant rarely takes down another.
RTO and RPO
Disaster recovery is defined by two numbers, and both are business decisions that engineering implements.
Recovery time objective (RTO) is the maximum acceptable time between the disaster and service restoration. Recovery point objective (RPO) is the maximum acceptable amount of data loss, expressed as time: an RPO of fifteen minutes means losing the last fifteen minutes of writes is tolerable.
Derive them from impact rather than aspiration. Ask what the organisation loses per hour of outage — revenue, regulatory penalties, contractual credits, reputational cost — and what it costs to re-enter lost transactions. Plot cost of downtime against cost of protection; the sensible target sits near their intersection, not at zero. Asking a business owner "how much downtime is acceptable?" always returns "none", so ask instead "what would you do for four hours without this?"
Tier the estate. A payment authorisation path might need an RTO of minutes and an RPO near zero; an internal reporting tool can accept a day of each. Applying the strictest tier everywhere is the most reliable way to make disaster recovery unaffordable and therefore unbuilt.
Redundancy patterns for high availability
Within a region, the standard building blocks are well understood.
Stateless tiers across zones. Put instances in at least three zones behind a load balancer, in an autoscaling group whose minimum survives losing one zone. Size each zone so the survivors absorb the failed zone's share — with three zones that is about 50% headroom, against the 100% two zones require.
Health checks that mean something. A check returning 200 whenever the process is alive keeps routing traffic to an instance that cannot reach its database; a check failing whenever any dependency is unreachable can remove the whole fleet during a transient blip. Use a shallow check for the load balancer and a deeper one for alerting and deployment gates.
Timeouts, retries with jitter, and circuit breakers. Retries without exponential backoff and randomised jitter convert a brief dependency slowdown into a self-inflicted denial of service. A circuit breaker that opens after a failure threshold and lets a trickle of probes through gives the dependency room to recover. Bulkheads — separate connection pools per dependency — stop one slow downstream service exhausting the resources every other request needs.
Graceful degradation. Decide in advance what the system does without each dependency. Serving stale cached prices, queueing writes for later processing, or disabling recommendations while keeping checkout alive are all better outcomes than an error page, and they must be designed, not improvised.
Managed multi-zone data services. Object storage and managed queues are already zone-redundant. Relational databases typically offer a synchronous standby in a second zone with automatic failover, plus asynchronous read replicas. Know which of your stores replicates synchronously — that is the difference between zero and non-zero data loss.
The four disaster-recovery strategies
Across regions, four patterns cover almost every requirement. Cost rises with each step, and recovery time falls.
Backup and restore. Backups and snapshots are copied to a second region; nothing runs there until needed. Infrastructure is recreated from templates and data restored from the latest copy. RPO equals the backup interval, RTO is hours to a day, and the standing cost is storage only. Correct for most internal systems, and the only strategy that also protects against logical corruption — which is why every other strategy includes it too.
Pilot light. The data layer is continuously replicated to the second region and kept live, while the compute tier exists as templates and images but runs at zero or near-zero capacity. Recovery means scaling up compute and promoting the replica. RPO is seconds to minutes, RTO tens of minutes, and the standing cost is the replicated data plus a small always-on footprint.
Warm standby. A complete but scaled-down copy of the environment runs continuously in the second region, serving no traffic or a token share. Failover is a traffic shift plus a scale-out. RPO is seconds, RTO minutes, and cost is materially higher — commonly a quarter to a half of production, depending on how far down it is scaled. The decisive advantage is that the standby is running, so its deployments, certificates and permissions are exercised continuously rather than discovered at 3 a.m.
Multi-site active-active. Both regions serve production traffic behind a global routing layer. RTO approaches zero, RPO depends on the replication model, and cost is at least double. It is the only option that also gives regional latency benefits, and it forces you to solve conflict resolution, global identity and simultaneous schema compatibility. The complexity is the real price; the infrastructure bill is the smaller part.
| Strategy | RPO | RTO | Standing cost | Main risk |
|---|---|---|---|---|
| Backup and restore | Hours | Hours to a day | Storage only | Restore never tested at full scale |
| Pilot light | Minutes | Tens of minutes | Data replication + minimal compute | Capacity unavailable when everyone fails over |
| Warm standby | Seconds | Minutes | Significant fraction of production | Drift between regions if not deployed to |
| Active-active | Seconds or zero | Near zero | Double or more | Conflicts, and complexity nobody fully models |
Database failover dominates the timeline
In nearly every recovery, the stateless tier is trivial and the database sets the clock.
Synchronous replication commits on the primary only once a standby has acknowledged the write: zero data loss, at a latency cost proportional to distance. It is practical between zones and generally impractical between distant regions. Asynchronous replication acknowledges locally and ships changes afterwards, so cross-region RPO equals the replication lag — which must be alarmed, because lag grows silently under write bursts and long transactions.
Promotion is the dangerous moment. Automatic cross-region promotion risks split brain: a partition convinces the secondary that the primary is dead while the primary keeps accepting writes, leaving two divergent histories. Within a region, quorum-based automatic failover is safe and standard; across regions most organisations require a human decision, because the failure signal is ambiguous.
Practical details decide whether promotion works: applications must reconnect rather than cache a resolved address forever; connection strings should point at a DNS alias or a proxy endpoint you can repoint; failback must be planned, because reversing replication is often harder than the failover; and the secondary must hold the same encryption keys, credentials and schema version. The consistency implications of accepting writes in two places are discussed in distributed transactions and consistency.
Backups: the part everyone claims and few verify
Follow a 3-2-1 discipline adapted for cloud: at least three copies, on two kinds of storage, with one isolated — a different account or subscription, different credentials, object lock or immutability enabled, and preferably a different region. The isolation is not about hardware failure; it is about an attacker or a mistaken automation holding delete permissions.
Then test restores on a schedule and measure the time, because a multi-terabyte restore from cold storage can exceed the RTO by an order of magnitude and only trying reveals it. Automate a periodic restore into a scratch environment, run a checksum and a business query against it, and record the elapsed time as a metric with an alarm on regression.
Proving it with game days
An untested recovery plan is a document, not a capability. A game day is a scheduled exercise in which the team executes a failure scenario against a real environment.
Start in a pre-production environment that genuinely mirrors production, announce the exercise, and state a hypothesis first: "if we terminate the primary database instance, traffic recovers within ninety seconds with no failed writes". Then inject the failure and record what actually happened. The valuable output is the list of surprises — a missing IAM role, an alert routed to a disbanded team, a runbook step naming a decommissioned tool.
Scenarios worth rehearsing, roughly in order of maturity: terminate an instance; fail over the database; lose an availability zone; restore the largest database and time it; expire a certificate; lose the identity provider; execute a regional failover; and recover from logical corruption discovered six hours late. That last one is rarely rehearsed and among the most likely to occur.
Automated fault injection makes this continuous rather than annual. Both major clouds offer managed fault-injection services that stop instances, add latency or throttle APIs within a controlled blast radius, with a stop condition tied to an alarm. Running small experiments continuously — first in pre-production, eventually in production with people watching — turns claimed resilience into measured resilience, and it is exactly the evidence a Well-Architected review asks for.
Key takeaways
- Availability targets are arithmetic: serial dependencies multiply, four nines demands automatic recovery, and a provider SLA is a credit schedule rather than a promise about your system.
- Redundancy only helps across independent failure domains; correlated failures from shared dependencies, control planes, configuration pushes and capacity shortfalls defeat naive duplication.
- RTO and RPO come from business impact and should be tiered per workload; applying the strictest tier everywhere makes disaster recovery unaffordable.
- The four DR strategies trade cost against recovery time — backup and restore, pilot light, warm standby, active-active — and every one of them still needs immutable, isolated backups.
- Database replication and promotion set the recovery timeline; synchronous within a region, asynchronous across regions, with failback planned in advance.
- A recovery capability exists only once it has been exercised: schedule game days, measure restore times, and inject faults continuously rather than trusting the runbook.
Sources and further reading
Frequently asked questions
- Does a provider's 99.99% SLA mean my application is 99.99% available?
- No, for two reasons. An SLA is a billing commitment — it promises service credits, not uptime — and it covers one component measured the provider's way. Your availability is the composite of every component in the request path plus your own code, deployments and configuration changes. A system built from four independent 99.99% services in series cannot exceed roughly 99.96% before you write a line of code.
- If we replicate the database continuously, do we still need backups?
- Yes, and the distinction is what replication does with a mistake. Replication faithfully copies a DELETE without a WHERE clause, a bad migration or a ransomware encryption to every replica within seconds. Backups and point-in-time snapshots preserve earlier states, which is the only defence against logical corruption. Keep at least one copy immutable and in a separate account or subscription so a compromised administrator cannot delete it.
- Is active-active across two regions worth it?
- Only when the loss from an hour of downtime clearly exceeds the sustained cost, and when the data model tolerates it. Active-active roughly doubles infrastructure spend, adds cross-region data transfer charges, and forces you to solve write conflicts, global session state and schema changes that must be compatible in both regions simultaneously. Most systems get better value from a well-tested warm standby than from an active-active design nobody fully understands.
- Why do failovers so often fail when they are genuinely needed?
- Because the untested parts activate at once. DNS records with long TTLs, capacity in the secondary region that was never reserved and is now unavailable because everyone else is failing over too, IAM roles that exist only in the primary, a runbook referencing a bastion host that has been decommissioned. Scheduled game days convert those from incident-time discoveries into ordinary Tuesday tickets.