Art of Codeartofcode
Lesson 6 of 16

11. Object / Blob Storage

1
The picture to keep in your head
2
11.1 What's the difference between block, file and object storage?
3
11.2 Why not just store files in the database?
4
11.3 What are buckets, keys and metadata?
5
11.4 How do uploads work with presigned URLs?
6
11.5 How do you upload very large files reliably?
7
11.6 How does object storage avoid losing files?
8
11.7 What are storage classes and lifecycle rules?
9
11.8 How should files be served to users?
10
11.9 What happens after a file is uploaded?
11
Memory card and where the analogy breaks
12
Practice: five levels
13
Interview questions
14
Solutions
15
Sources
16
Code lab
11. Object / Blob Storage

11.5 How do you upload very large files reliably?

3 min read+10 XP

At the cloakroom: A guest arrives with fifty boxes of wedding gifts. Carrying them all in one trip is impossible, and if they drop the pile at the last step, they start over. Instead, they bring the boxes a few at a time, any order, and if one box falls, they just carry that one again. The attendant stacks them in order at the end.

In the system: Multipart upload splits a large file into parts that are uploaded separately, then joined.

a 5 GB video uploaded in 50 parts; a failed part is retried alone; storage joins parts at the end.

Figure: a dropped connection costs one 100 MB part, not the whole 5 GB.

  1. Start a multipart upload; storage returns an upload ID.
  2. Upload parts (each with a part number), in parallel and in any order. Retry any part that fails.
  3. Complete: storage joins the parts in part-number order into one object.

Why it matters:

  • Reliability: a 5 GB upload over a 20 Mbps connection takes about 33 minutes. Failing at minute 30 without multipart means starting over; with multipart, you redo one part.
  • Speed: parallel parts use more of the connection.
  • Limits: on S3, parts are 5 MiB to 5 GiB, up to 10,000 parts, for objects up to about 48.8 TiB, and AWS recommends multipart for objects of about 100 MB and up.

Clean up unfinished uploads: abandoned parts take space and cost money. A lifecycle rule (11.7) can delete them after a few days.

Memory code: "Many small boxes, any order; drop one, carry one again."