Design a Distributed Job Scheduler: Interview Walkthrough

13 min read · Last updated: October 24, 2026

To design a distributed job scheduler, split it into three parts: a scheduler that turns schedules into run records, a durable queue that holds those runs, and a pool of stateless workers that claim and execute them. Delivery is at-least-once, so every job must be idempotent, and every run gets a unique key.

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The specs

• RFC 9110, HTTP Semantics, section 15.3.3 "202 Accepted" (June 2022)
• Google AIP-151, Long-running operations

The docs (all checked October 2026)

• PostgreSQL, SELECT locking clause (SKIP LOCKED)
• PostgreSQL, explicit locking and advisory locks
• Kubernetes, CronJob
• Kubernetes, Leases
• Celery, periodic tasks (beat)
• Celery, tasks (retries, acks_late)
• Celery, backends and brokers
• RabbitMQ, consumer acknowledgements and prefetch
• RabbitMQ, consumers (delivery timeout)
• Amazon SQS, visibility timeout
• Amazon SQS, exactly-once processing in FIFO queues
• KEDA, RabbitMQ queue scaler

The essays

• Martin Kleppmann, "How to do distributed locking", February 8, 2016
• Discord Engineering, "How Discord Stores Billions of Messages", January 13, 2017

What you'll build and why

By the end of this walkthrough you will have a full interview answer for "design a distributed job scheduler", plus working SQL for the core of it. Think of the system behind "send this report every Monday at 9am" or "resize this video when it's uploaded".

The answer covers seven things, in the order I would say them out loud:

1. Requirements and the one guarantee that matters
2. The schema
3. The scheduler loop that turns cron into rows
4. The queue and how workers claim jobs
5. Leader election, and why it should not be load-bearing
6. Retries and the exactly-once trap
7. An API for long-running tasks

Why this question? Because it tests almost every distributed systems idea in one box: locking, failure, duplicates, scaling, and API design. If you have ever lost an interview by drawing boxes without saying why, read why most developers fail system design interviews first. This walkthrough is the "why" for one specific question.

Key takeaways

• Kubernetes says a CronJob creates a Job "approximately once", and that two Jobs or none can be created. Real schedulers are not exactly-once.
• The fix is a unique key on (job_id, scheduled_for) plus idempotent job code. That one constraint makes duplicate scheduling harmless.
• PostgreSQL's FOR UPDATE SKIP LOCKED lets many workers pull from one table without blocking each other.
• Amazon SQS hides a received message for 30 seconds by default, up to a 12-hour maximum. A worker that dies just lets the message reappear.
• Celery retries 3 times by default and caps exponential backoff at 600 seconds.
• Long-running work should return HTTP 202 Accepted with a status URL, not hold the connection open.

Prerequisites

You do not need to have built a scheduler before. You do need:

• Basic SQL: INSERT, UPDATE, transactions, unique constraints.
• A rough idea of what a message queue is (a producer puts a message in, a consumer takes it out).
• Comfort with the phrase "the process might die at any line". That is the whole interview.

The code below uses PostgreSQL, because one database you already know is easier to defend than three you don't. I will point out where RabbitMQ or Amazon SQS fit.

How to design a distributed job scheduler in an interview

Start with requirements, then a schema, then the scheduler loop, the queue, and the workers. Say out loud that delivery is at-least-once, so jobs must be idempotent. Then cover leader election, retries, and scaling. Interviewers reward the candidate who names the failure modes before being asked.

Here is the step by step answer.

Step 1: Pin down the requirements

Ask before you draw. These are the questions that change the design:

Question to ask	Why it changes the design
One-off jobs, recurring (cron), or both?	Recurring needs a scheduler loop that creates runs.
How late can a job start? Seconds or minutes?	Decides polling interval and whether you need a push queue.
How long does a job run? Seconds or hours?	Long jobs need heartbeats and a status API.
Can a job run twice safely?	If not, you need idempotency keys in the downstream system.
How many jobs per second at peak?	Decides one database versus a dedicated broker.
Do jobs need ordering?	Ordering per key limits parallelism.

Then state your guarantee in one sentence: "Every scheduled run will execute at least once, and duplicate executions will be harmless." That sentence tells the interviewer you already know exactly-once is a trap.

Step 2: Design the schema

Separate the job definition from each run. A definition says what and when. A run is one attempt at one scheduled time.

CREATE TABLE jobs (
  id            BIGSERIAL PRIMARY KEY,
  name          TEXT NOT NULL,
  cron_expr     TEXT,                 -- NULL for one-off jobs
  timezone      TEXT NOT NULL DEFAULT 'UTC',
  payload       JSONB NOT NULL,
  next_run_at   TIMESTAMPTZ,
  enabled       BOOLEAN NOT NULL DEFAULT TRUE
);

CREATE TABLE job_runs (
  id            BIGSERIAL PRIMARY KEY,
  job_id        BIGINT NOT NULL REFERENCES jobs(id),
  scheduled_for TIMESTAMPTZ NOT NULL,
  status        TEXT NOT NULL DEFAULT 'queued', -- queued | running | succeeded | dead
  attempts      INT NOT NULL DEFAULT 0,
  max_attempts  INT NOT NULL DEFAULT 5,
  run_after     TIMESTAMPTZ NOT NULL DEFAULT now(),
  locked_by     TEXT,
  locked_until  TIMESTAMPTZ,
  last_error    TEXT,
  UNIQUE (job_id, scheduled_for)
);

CREATE INDEX job_runs_ready ON job_runs (run_after) WHERE status = 'queued';

The most important line is UNIQUE (job_id, scheduled_for). It means "the 9am run of job 42" can exist exactly once, no matter how many schedulers try to create it.

The partial index keeps the "what's ready?" query fast, because it only covers queued rows.

Step 3: Write the scheduler loop

The scheduler wakes up every few seconds, finds jobs that are due, creates a run for each, and moves next_run_at forward.

BEGIN;

WITH due AS (
  SELECT id, next_run_at
  FROM jobs
  WHERE enabled AND next_run_at <= now()
  FOR UPDATE SKIP LOCKED
  LIMIT 100
)
INSERT INTO job_runs (job_id, scheduled_for, run_after)
SELECT id, next_run_at, next_run_at FROM due
ON CONFLICT (job_id, scheduled_for) DO NOTHING;

-- then, in application code, compute the next cron time for each due job
-- in its own timezone and UPDATE jobs SET next_run_at = ... WHERE id = ...

COMMIT;

ON CONFLICT DO NOTHING is the safety net. If two schedulers fire at once, the second insert is silently dropped.

Store timezone per job and compute the next time in that zone. "Every day at 9am" in a zone with daylight saving is not every 24 hours. Kubernetes CronJob has a dedicated time zone field for this, and its docs say putting CRON_TZ or TZ inside the schedule string is not officially supported.

How does a distributed cron work?

A distributed cron stores schedules in shared storage instead of one machine's crontab. A scheduler (or several) computes the next run time, writes a run record, and hands it to a queue. Any worker can execute it. The schedule survives machine failure because it lives in the database, not in a process.

Compare that to single-machine cron. If that one box dies, nothing runs, and nobody notices until the Monday report is missing.

Two real systems show the trade-offs clearly:

• Kubernetes CronJob. Its docs say a CronJob creates a Job "approximately once per execution time", and that "two Jobs might be created, or no Job might be created". The docs conclude that your Jobs "should be idempotent". It also offers concurrencyPolicy with Allow (the default), Forbid and Replace, and startingDeadlineSeconds to skip runs that start too late. If more than 100 schedules were missed, it does not start the Job at all and logs an error.
• Celery beat. Celery's scheduler triggers tasks at set intervals for workers to pick up. Its docs are blunt: "You have to ensure only a single scheduler is running for a schedule at a time, otherwise you'd end up with duplicate tasks."

That Celery line is why Steps 4 and 5 exist.

How do workers claim jobs without running the same one?

Each worker atomically locks a batch of ready rows and skips rows another worker already holds. In PostgreSQL that is FOR UPDATE SKIP LOCKED. The worker also takes a short lease, so if it crashes, the lease expires and the job goes back to the queue for someone else to pick up.

Step 4: Build the queue and the workers

UPDATE job_runs
SET status = 'running',
    locked_by = $1,
    locked_until = now() + interval '5 minutes',
    attempts = attempts + 1
WHERE id IN (
  SELECT id FROM job_runs
  WHERE status = 'queued' AND run_after <= now()
  ORDER BY run_after
  LIMIT 10
  FOR UPDATE SKIP LOCKED
)
RETURNING id, job_id, scheduled_for, attempts;

The PostgreSQL docs say skipping locked rows "provides an inconsistent view of the data, so this is not suitable for general purpose work, but can be used to avoid lock contention with multiple consumers accessing a queue-like table." That is exactly our case.

locked_until is a lease. A worker that crashes stops renewing it. A small reaper puts expired leases back:

UPDATE job_runs
SET status = 'queued', locked_by = NULL, run_after = now()
WHERE status = 'running' AND locked_until < now();

Long jobs extend locked_until every minute or so, like a heartbeat.

Dedicated queues give you the same idea out of the box:

Concept	PostgreSQL table	RabbitMQ	Amazon SQS
Claim a job	FOR UPDATE SKIP LOCKED	Deliver to consumer	ReceiveMessage
Lease while working	locked_until column	Unacked delivery	Visibility timeout (30 s default, 12 h max)
Worker dies	Reaper requeues expired lease	Unacked messages "automatically requeued" when the channel closes	Message becomes visible again
Long job	Renew locked_until	Ack before the consumer timeout (30 min default)	ChangeMessageVisibility heartbeat
Gave up	status = 'dead'	Dead letter exchange	Dead-letter queue

My advice in an interview: say "I'd start with Postgres and SKIP LOCKED, and move to a broker when throughput demands it." It shows you can defend the simple version.

Step 5: Add leader election, but don't trust it

You probably want only one active scheduler, to avoid wasted work. Kubernetes does the same for its own kube-controller-manager and kube-scheduler: Lease objects ensure "only one instance of a component is running at any given time", while the others wait on standby.

With PostgreSQL you can use an advisory lock as a simple leader lock:

-- each scheduler instance runs this on startup and keeps the session open
SELECT pg_try_advisory_lock(424242);  -- true = you are the leader

The PostgreSQL docs say a session-level advisory lock is "held until explicitly released or the session ends", so a crashed leader's lock frees itself.

Now the senior-level point. A leader can pause (a long garbage collection, a frozen VM), lose its lease, and wake up still thinking it's the leader. Martin Kleppmann's essay walks through this exact case: client 1 holds token 33 and pauses, client 2 takes the lock with token 34 and writes, and client 1's late write with token 33 must be rejected. That number is a fencing token: it goes up with every new lock holder, and the storage layer checks it.

Our design already has a fence. The unique key on (job_id, scheduled_for) means two leaders produce one run, not two. Leader election saves effort. The unique constraint protects correctness. Say that sentence in the interview.

Step 6: Handle retries

Failures are normal. Retry with exponential backoff and jitter, and stop after a limit.

-- on failure
UPDATE job_runs
SET status = CASE WHEN attempts >= max_attempts THEN 'dead' ELSE 'queued' END,
    run_after = now() + (interval '1 second' * LEAST(600, power(2, attempts)) * random()),
    last_error = $2,
    locked_by = NULL
WHERE id = $1;

The delay is random, up to a cap that doubles with each attempt and never exceeds 600 seconds. The randomness stops a thousand failed jobs from retrying in the same second and knocking the downstream service over again.

Celery's defaults are a good reference point. max_retries defaults to 3, default_retry_delay is three minutes, and with retry_backoff on, retry_backoff_max defaults to 600 seconds and retry_jitter defaults to true.

How to avoid running a scheduled job twice

You cannot fully prevent a job from running twice, so make the second run harmless. Put a unique key on each scheduled run, pass that key to every side effect as an idempotency key, and record completion in the same transaction as the work. Duplicates then become no-ops instead of double charges.

Here is why "exactly-once" is a trap. A worker charges a card, then crashes before it acks the message. The queue cannot know the charge happened, so it redelivers. Every queue in this article says this out loud:

• Amazon SQS: "because of the at-least-once delivery model, Amazon SQS doesn't guarantee that a message won't be delivered more than once within the visibility timeout period."
• RabbitMQ: consumers "must be prepared to handle redeliveries", and redelivered messages carry redeliver = true.
• Celery: "Ideally task functions should be idempotent." Its acks_late setting acknowledges a message only after the task has run, which trades lost tasks for possible duplicates.

What about SQS FIFO queues and their "exactly-once processing"? Read the fine print. FIFO deduplication stops duplicate sends: if you retry SendMessage within the 5-minute deduplication interval, SQS doesn't add a duplicate. It does not stop your worker from crashing mid-job.

The practical pattern:

BEGIN;
INSERT INTO processed_runs (run_id) VALUES ($1);  -- primary key on run_id
-- do the database side effect here, in the same transaction
COMMIT;
-- a duplicate run fails the INSERT and does nothing

For external APIs, send the run ID as the provider's idempotency key, if it supports one. If you want the wider idea behind designing parts that don't need to coordinate, I wrote about it in distributed systems: independence over coordination.

How to build a long-running REST API with a queue

Accept the request, write a job row, push it to the queue, and return HTTP 202 Accepted immediately with a URL the client can poll. Workers update the job's status as they go. The client polls that URL, or gets a webhook, until the job is done and the result or error is available.

Step 7: Design the API for long-running tasks

RFC 9110, the HTTP Semantics spec from June 2022, says 202 means "the request has been accepted for processing", with processing not yet complete. It says the response should describe the request's status and point to a status monitor.

POST /v1/reports
Content-Type: application/json
Idempotency-Key: 7f3c9a

{"month": "2026-09"}

HTTP/1.1 202 Accepted
Location: /v1/operations/op_81f2

{"name": "op_81f2", "done": false}

GET /v1/operations/op_81f2

HTTP/1.1 200 OK

{"name": "op_81f2", "done": true,
 "metadata": {"progress": 100},
 "response": {"url": "/v1/reports/2026-09.csv"}}

The response shape borrows from Google's AIP-151. An operation has a name, a done flag, metadata for progress, and then either a response or an error. AIP-151 recommends this pattern for methods that take significant time, with about 10 seconds as its rule of thumb, and says finished operations may expire, with 30 days as a rule of thumb.

The Idempotency-Key header matters here too. A client that times out and retries the POST should get the same operation back, not a second report.

A quick tangent on status codes: for internal service-to-service APIs, some teams skip HTTP status semantics entirely. I made that case in why your private API doesn't need HTTP status codes. For a public async API, 202 plus a status URL is the clear, standard choice.

How to scale workers for a job queue

Keep workers stateless, then scale them on queue depth, not CPU. A tool like KEDA, the Kubernetes event-driven autoscaler, adds worker pods as messages pile up. Tune prefetch so each worker only holds what it can process, and split queues by job type so one slow job type doesn't starve the rest.

KEDA's RabbitMQ scaler shows the idea. In QueueLength mode, value is the target number of messages per instance. The docs' example: a value of 10 with 30 queued messages scales to 3 pods.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: job-workers
spec:
  scaleTargetRef:
    name: job-worker-deployment
  triggers:
  - type: rabbitmq
    metadata:
      protocol: amqp
      queueName: jobs
      mode: QueueLength
      value: "20"
    authenticationRef:
      name: keda-trigger-auth-rabbitmq-conn

Prefetch is the other knob. RabbitMQ's docs say prefetch values "in the 100 through 300 range usually offer optimal throughput", while a prefetch of 1 "is the most conservative". For short jobs, go high. For hour-long jobs, use 1, or one worker hoards messages that idle workers could have taken.

Separate queues by job type (fast emails, slow video encodes) so a burst of slow jobs doesn't block fast ones. With Postgres, the claim query stays fast as long as the partial index covers only queued rows and you archive finished runs.

Celery vs RabbitMQ for background jobs

This is not an either-or. RabbitMQ is a message broker: it stores and delivers messages. Celery is a Python task framework that sits on top of a broker and adds task definitions, retries, scheduling (beat) and result storage. You often run Celery with RabbitMQ as its broker.

	RabbitMQ	Celery
What it is	Message broker	Task queue framework for Python
Language	Any language with an AMQP client	Python
Retries and backoff	You build them (dead letter exchanges, requeue)	Built in (autoretry_for, retry_backoff)
Periodic jobs	Not built in	Celery beat
Needs a broker?	It is the broker	Yes: RabbitMQ, Redis and Amazon SQS are listed as stable

Celery's own docs note RabbitMQ "handles larger messages better than Redis", while Redis "works well for rapid transport of small messages". If you write Python and want retries and cron without building them, use Celery. If you have workers in several languages, talk to the broker directly.

Common mistakes

These are the mistakes I'd watch for, in interviews and in production:

1. Promising exactly-once. Say at-least-once plus idempotency instead. Every queue doc cited here says the same.
2. Running two Celery beat or cron instances "for HA". Without a unique run key, that doubles every job.
3. Making leader election the only guard. A paused leader will wake up and write. Fence at the storage layer.
4. A visibility timeout shorter than the job. On SQS the message reappears and a second worker starts the same job. Heartbeat with ChangeMessageVisibility.
5. Retrying without jitter. Synchronised retries hit a recovering service all at once.
6. No dead-letter state. A poison job that always fails will retry forever and hide real failures.
7. Ignoring time zones. Store the zone per job, and compute the next run in that zone.
8. Holding an HTTP connection open for a long job. Return 202 and a status URL.

Follow-up questions interviewers ask

After the main design, expect a pivot. These are the common ones, with a one-line answer for each:

• "What if the database is the bottleneck?" Move the hot queue to a broker like RabbitMQ or SQS, and keep Postgres as the source of truth for definitions and history.
• "How do you run a job at an exact second?" You can't promise it. Offer a start deadline, like Kubernetes' startingDeadlineSeconds, and skip runs that are too late.
• "What if a run takes longer than its interval?" Pick a policy: allow overlap, skip (Kubernetes Forbid), or replace (Kubernetes Replace).
• "How would you store chat messages?" This one shows up because it tests the same skill: choosing keys for your hottest query.

How to store chat messages in a database (the system design follow-up)

Partition by conversation and time, and sort by a time-ordered ID. Discord's 2017 engineering post describes the design it used after moving from MongoDB to Cassandra: a primary key of ((channel_id, bucket), message_id), where each bucket holds about 10 days of messages to keep partitions under 100MB.

Discord uses Snowflake IDs, which are chronologically sortable, so message_id doubles as the sort key and "load the latest 50 messages" becomes a single range read. At the time of that post, Discord was storing more than 120 million messages a day.

CREATE TABLE messages (
  channel_id BIGINT,
  bucket     INT,
  message_id BIGINT,
  author_id  BIGINT,
  content    TEXT,
  PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

The lesson carries back to schedulers: pick your key to match your hottest query.

Next steps

To make this stick, build the Postgres version in an afternoon:

1. Create the two tables above.
2. Write a scheduler loop that inserts runs with ON CONFLICT DO NOTHING.
3. Start three worker processes running the SKIP LOCKED claim query.
4. Kill a worker mid-job with kill -9 and watch the reaper requeue it.
5. Run two schedulers at once and confirm you still get one run per time slot.

Once you have watched duplicates fail to happen, you will explain this design with far more confidence than someone who only memorised a diagram.

FAQ

Is exactly-once delivery possible in a job scheduler?

Not end to end. Queues such as Amazon SQS and RabbitMQ deliver at least once, and a worker can crash after doing the work but before acknowledging it. What you can build is effectively-once: at-least-once delivery plus idempotent jobs keyed by a unique run ID.

Should I use a database or a message broker as the job queue?

Start with your database if load is modest. PostgreSQL's FOR UPDATE SKIP LOCKED is documented for multiple consumers on a queue-like table, and it keeps job state and history in one place. Move to RabbitMQ or SQS when throughput or fan-out outgrows a single database.

What is a fencing token?

A fencing token is a number that increases every time a new client gets a lock. The client sends it with every write, and the storage layer rejects any write with an older token. It protects you when a paused leader wakes up after losing its lock.

How often should workers poll for jobs?

It depends on how late a job is allowed to start. If a few seconds of delay is fine, a database-backed queue with a short sleep when nothing is ready works well. If you need lower latency, use a broker that pushes deliveries to consumers instead of polling.

Build the Postgres version this week, and use the free tools on Art of Code for the JSON and cron fiddling along the way. If you are preparing for interviews right now, browse open roles on the Art of Code jobs board.

Written by Adithya Guttha, Founder of Art of Code.