Art of Codeartofcode
Lesson 12 of 18

29. Failures, Retries, Circuit Breakers

1
The picture to keep in your head
2
29.1 What kinds of failure are there?
3
29.2 What are single points of failure and redundancy?
4
29.3 How does the "nines" math work?
5
29.4 Why do you need timeouts?
6
29.5 How do retries with exponential backoff and jitter work?
7
29.6 What is a retry storm?
8
29.7 What is a circuit breaker?
9
29.8 What are bulkheads?
10
29.9 What is graceful degradation?
11
29.10 How do health checks and failover work?
12
29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?
13
Memory card and where the analogy breaks
14
Practice: five levels
15
Interview questions
16
Solutions
17
Sources
18
Code lab
29. Failures, Retries, Circuit Breakers

29.11 What is disaster recovery, in brief (RPO, RTO, multi-region)?

3 min read+10 XP

At home: Insurance and a spare set of keys at a relative's house. If the house floods, how much do you lose (everything since the last backup of your photos?), and how long until you're living normally again?

In the system: Disaster recovery (DR) plans for losing a whole data centre or region. Two numbers define it:

  • RPO (Recovery Point Objective): how much data you can afford to lose, measured in time. With asynchronous replication running 2 s behind and 50 orders a second, losing the primary region can lose about 100 orders (computed in the chapter's code).
  • RTO (Recovery Time Objective): how long you can afford to be down.

a timeline from last good copy to disaster to recovery, and four DR strategies from backup-and-restore to multi-region active.

Figure: lower RPO and RTO cost more; choose per system, not one answer for everything.

Common strategies, cheapest first:

  • Backup and restore: regular backups to another region; RPO and RTO in hours.
  • Pilot light: data continuously replicated to another region, servers off until needed; RPO minutes, RTO tens of minutes.
  • Warm standby: a small, running copy in another region, scaled up on failover; RPO seconds, RTO minutes.
  • Multi-region active-active: both regions serve traffic; near-zero RTO and very low RPO, at the highest cost and complexity (conflict handling, Chapter 16).

Whatever you choose:

  • Backups you've never restored aren't backups. Test restores regularly.
  • Practise failover (game days, chaos experiments) so it works under pressure.
  • Different tiers: payments and orders may need minutes; analytics can wait a day.

Memory code: "Decide how much you can lose and how long you can wait, then practise the move."