Art of Codeartofcode
Lesson 7 of 16

11. Object / Blob Storage

1
The picture to keep in your head
2
11.1 What's the difference between block, file and object storage?
3
11.2 Why not just store files in the database?
4
11.3 What are buckets, keys and metadata?
5
11.4 How do uploads work with presigned URLs?
6
11.5 How do you upload very large files reliably?
7
11.6 How does object storage avoid losing files?
8
11.7 What are storage classes and lifecycle rules?
9
11.8 How should files be served to users?
10
11.9 What happens after a file is uploaded?
11
Memory card and where the analogy breaks
12
Practice: five levels
13
Interview questions
14
Solutions
15
Sources
16
Code lab
11. Object / Blob Storage

11.6 How does object storage avoid losing files?

3 min read+10 XP

At the cloakroom: A careful hall keeps coats in separate buildings, so a fire in one doesn't burn everything. It could keep three identical copies of every coat (expensive), or something cleverer: tear each coat's description into pieces plus a few spare pieces, so the coat can be rebuilt from most of them.

In the system: Object stores promise very high durability (the chance a stored object is never lost). AWS, for example, says S3 Standard is designed for 99.999999999% ("eleven nines") durability, storing data across multiple availability zones (separate data centres in a region). Two techniques do the work:

replication keeps three full copies; erasure coding splits data into 10 data and 4 parity pieces.

Figure: replication is simple but triples storage; erasure coding survives the same kinds of loss with far less extra space.

  • Replication: keep full copies on different machines and zones. Simple and fast to read, but 3 copies means 3× the storage.
  • Erasure coding: split each object into, for example, 10 data pieces, compute 4 extra parity pieces, and store all 14 on different machines. Any 10 of the 14 can rebuild the object, so it survives losing any 4, for only 1.4× the storage. Rebuilding costs some CPU, and reads may touch several machines.

Both run with constant checksums: the store regularly checks pieces against their fingerprints and repairs any that have quietly gone bad.

Durability isn't backup. If your code deletes a file or overwrites it, eleven nines faithfully keeps the wrong version. Protect against your own mistakes with versioning (keep old versions of each object), restricted delete permissions and, for critical data, copies in another region or account.

Also note consistency: S3 has given strong read-after-write consistency since December 2020, so a read right after a successful write returns the new data. Older guides that warn otherwise are out of date for S3; check your provider's docs.

Memory code: "Copies cost three times; pieces cost far less. Neither saves you from your own delete."