39. Design File Storage and Sync (Google Drive / Dropbox)
1
The interview prompt2
39.1 What do you ask first?3
39.2 How big is this, really?4
39.3 What are the doors?5
39.4 What do we store?6
39.5 What does the whole system look like?7
39.6 How does chunking work?8
39.7 How does de-duplication work?9
39.8 How do sync and conflicts work?10
39.9 How is the metadata database designed?11
39.10 How do presigned uploads work?12
39.11 What would you say in the last five minutes?13
Score yourself14
Variations to practise15
Sources16
Code lab39. Design File Storage and Sync (Google Drive / Dropbox)
39.6 How does chunking work?
3 min read+10 XP
Step 5: Deep dives (10 minutes)
Splitting files into blocks lets you resume big transfers, upload only what changed, and store identical blocks once. Dropbox has used chunks of up to 4 MB, hashed with SHA-256.
Figure: fixed boundaries shift with every insertion; content-defined boundaries move with the content.
- Fixed-size chunks (every 4 MB): simple and fast. But inserting bytes near the start shifts every boundary, so every chunk's hash changes. In the chapter's code, adding 15 bytes to the start of a 200 KB file makes all 49 fixed chunks new.
- Content-defined chunking (CDC): slide a rolling hash over the bytes and cut wherever it matches a pattern, with a minimum and maximum size. Cut points depend on the content, so after an insertion they line up again. In the chapter's code, the same edit changes 1 of 35 chunks.
Our pick: fixed-size blocks are fine for most files (photos, videos and PDFs are rarely edited in the middle); content-defined chunking helps for large files edited in place, like documents and databases. Some systems use both: fixed blocks for storage and a smarter delta for transfer.
Memory code: "Cut by content, not by ruler, and an edit only changes its neighbours."