Art of Codeartofcode
Lesson 10 of 16

39. Design File Storage and Sync (Google Drive / Dropbox)

1
The interview prompt
2
39.1 What do you ask first?
3
39.2 How big is this, really?
4
39.3 What are the doors?
5
39.4 What do we store?
6
39.5 What does the whole system look like?
7
39.6 How does chunking work?
8
39.7 How does de-duplication work?
9
39.8 How do sync and conflicts work?
10
39.9 How is the metadata database designed?
11
39.10 How do presigned uploads work?
12
39.11 What would you say in the last five minutes?
13
Score yourself
14
Variations to practise
15
Sources
16
Code lab
39. Design File Storage and Sync (Google Drive / Dropbox)

39.9 How is the metadata database designed?

3 min read+10 XP

Metadata is small per file but huge in total (100 billion rows) and needs transactions (a commit writes a version and a journal entry together).

  • Shard by namespace (a user's root or a shared folder): all of a folder's files, versions and journal live on one shard, so commits and "changes since cursor" are single-shard operations (Chapter 15).
  • A relational database per shard (MySQL or PostgreSQL), with replicas (Chapter 14). Dropbox, for example, has written about building its metadata on sharded MySQL.
  • Hot namespaces: a huge shared company folder can outgrow a shard; split it into sub-namespaces, or give it a dedicated shard.
  • Moves and renames are metadata-only: change paths, never touch blocks.
  • Caching the latest file list per namespace speeds up listing, invalidated by journal entries.

Memory code: "Shard by folder, transact inside it, never touch bytes for a rename."