Art of Codeartofcode
Lesson 8 of 16

39. Design File Storage and Sync (Google Drive / Dropbox)

1
The interview prompt
2
39.1 What do you ask first?
3
39.2 How big is this, really?
4
39.3 What are the doors?
5
39.4 What do we store?
6
39.5 What does the whole system look like?
7
39.6 How does chunking work?
8
39.7 How does de-duplication work?
9
39.8 How do sync and conflicts work?
10
39.9 How is the metadata database designed?
11
39.10 How do presigned uploads work?
12
39.11 What would you say in the last five minutes?
13
Score yourself
14
Variations to practise
15
Sources
16
Code lab
39. Design File Storage and Sync (Google Drive / Dropbox)

39.7 How does de-duplication work?

3 min read+10 XP

The block store is content-addressed: a block's key is its SHA-256 hash.

  • Before uploading, the client asks "which of these hashes are missing?" and uploads only those. In the chapter's code, a second user saving the same file uploads nothing, and an edited file adds about one chunk, not 200 KB.
  • Across users: the same installer or PDF in a million accounts is stored once. Per-user encryption keys limit this (identical plaintext gives different ciphertext), so many services de-duplicate within an account or organisation.
  • Privacy caution: cross-user de-duplication can leak that a file already exists (an upload that "finishes instantly" reveals someone else has it). Researchers demonstrated this kind of side channel against Dropbox's early design; limiting de-duplication to a user's own data, or always uploading for small files, closes it.

Memory code: "The hash is the name: if we have it, don't send it."