Avoid Duplicate Messages in Kafka: Idempotent Consumers

18 min read · Last updated: October 24, 2026

To avoid duplicate messages in a Kafka consumer, make the consumer idempotent: give every message a unique ID, record that ID in a dedup table in the same database transaction as your side effect, and commit the Kafka offset only after that transaction commits. A redelivered message then hits a primary key conflict and gets skipped.

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The Kafka docs

• Apache Kafka design docs: Message Delivery Semantics (Kafka 4.2)
• Apache Kafka producer configs: enable.idempotence (Kafka 4.2)
• Apache Kafka consumer configs: enable.auto.commit (Kafka 4.2)
• Apache Kafka introduction: keys, partitions and ordering (Kafka 4.2)
• KIP-679: Producer will enable the strongest delivery guarantee by default

The SQS and RabbitMQ docs

• Amazon SQS at-least-once delivery
• Amazon SQS queue types: standard vs FIFO
• Amazon SQS exactly-once processing (FIFO)
• Amazon SQS visibility timeout
• RabbitMQ Reliability Guide
• RabbitMQ broker semantics: message ordering
• RabbitMQ consumers: Single Active Consumer

The patterns and specs

• Idempotent Consumer pattern, microservices.io
• Transactional Outbox pattern, microservices.io
• Stripe API: Idempotent requests
• IETF draft: The Idempotency-Key HTTP Header Field

The benchmark

• Confluent: Exactly-once Semantics Are Possible, Neha Narkhede and Guozhang Wang, June 30 2017 (updated March 2025)

Key takeaways

• Apache Kafka, the open-source event streaming platform, defaults to at-least-once delivery. Duplicates are part of the contract, not a bug.
• Amazon SQS standard queues and RabbitMQ with acknowledgements are also at-least-once. All three sets of official docs tell you to make consumers idempotent.
• The fix is a dedup table with a primary key on the message ID, written in the same transaction as the business change.
• The idempotent Kafka producer is on by default (enable.idempotence=true) in current releases, but it only stops duplicates in the log. It does nothing for your consumer.
• SQS FIFO deduplicates sends only inside a 5-minute window. Kafka transactions cost about 3% throughput in Confluent's benchmark, and neither covers your database or an email you sent.
• The outbox pattern fixes the "saved to the DB but never published" problem, and its relay can still publish twice. Your consumer must still be idempotent.

What you'll build, and why

You'll build a Kafka consumer in Python that charges an order once in effect, even when Kafka hands it the same message two or three times. Then you'll add the outbox pattern on the producer side so the event is never lost either.

Here is why this matters. Picture a payment consumer that reads OrderPlaced, charges the card, and then commits its offset. The pod gets killed between the charge and the commit. Kafka has no record that you finished, so the next consumer in the group reads the same message and charges the card again.

That is not a rare edge case. It is the exact scenario the Kafka design docs describe: the consumer "crashes after processing messages but before saving its position," and the process that takes over handles some messages a second time.

You cannot configure this away. You design around it.

What is an idempotent consumer?

An idempotent consumer is a message handler that produces the same end result whether it processes a message once or ten times. It usually works by recording each processed message ID in a database table with a unique constraint, inside the same transaction as the business change, so a repeat delivery is detected and skipped.

The definition comes from the Idempotent Consumer pattern on microservices.io. The handler inserts the message ID into a PROCESSED_MESSAGE table keyed on (subscriberId, messageID). A duplicate makes that insert fail, so the handler rolls back and ignores the message.

Some operations are idempotent by nature. "Set order 42 to SHIPPED" can run twice safely. The Kafka docs make the same point: reprocessing an update keyed by a primary key just overwrites a record with an identical copy.

Other operations are not. "Subtract $50 from the balance" or "send the welcome email" gives a wrong result on every repeat. Those are the ones that need the dedup table.

At-least-once vs exactly-once delivery: what's the difference?

At-most-once means a message may be lost but is never redelivered. At-least-once means a message is never lost but may be redelivered. Exactly-once means each message is processed once and only once. Kafka, SQS standard queues and RabbitMQ all give you at-least-once by default, so your code must handle duplicates.

These three definitions are straight from the Kafka design docs. The trade-off is simple: you pick between losing messages and seeing them twice. Almost everyone picks seeing them twice, because a duplicate can be detected and a lost message cannot.

Here is how the brokers compare, using only their own documentation.

Broker	Default delivery	Built-in dedup	Ordering guarantee	What the docs tell you to do
Apache Kafka	At-least-once	Idempotent producer stops duplicate log writes on retry	In order within one partition; same key goes to the same partition	Store offset with output, or make processing idempotent
Amazon SQS standard	At-least-once	None	Best-effort	"Design your applications to be idempotent"
Amazon SQS FIFO	Exactly-once processing	Dedup ID or content hash, 5-minute window	Strict within a message group	Set MessageDeduplicationId or enable content-based dedup
RabbitMQ (with acks)	At-least-once	None; redelivered flag is a hint	In order per channel and queue	Design the consumer to be idempotent

Does SQS deliver messages more than once?

Yes. Amazon SQS standard queues are at-least-once. AWS stores copies of each message on multiple servers, and if one server is unavailable when you delete a message, its copy survives and can be delivered again. AWS's own advice is to design your application to be idempotent. FIFO queues avoid duplicates, but only for sends retried within 5 minutes.

There is a second path to duplicates in SQS: the visibility timeout. When you receive a message it stays in the queue, hidden for 30 seconds by default. If you don't delete it before the timeout expires, it becomes visible again and another consumer can pick it up. A slow handler plus a short timeout equals a duplicate.

AWS is blunt about it in the visibility timeout docs: because of the at-least-once model, there is "no absolute guarantee that a message won't be delivered more than once during the visibility timeout period."

FIFO queues help, with a catch. According to the AWS docs, they deduplicate when you retry SendMessage within a 5-minute interval, using either your MessageDeduplicationId or a SHA-256 hash of the message body. They also cap throughput at 300 API calls per second per method without batching, or 3,000 messages per second with batches of 10. And the 5-minute dedup does nothing if your consumer charges a card and then crashes before deleting the message.

Prerequisites

You need:

• Python 3.10 or newer
• A running Kafka broker and a topic called orders
• PostgreSQL, the open-source relational database
• pip install confluent-kafka "psycopg[binary]"

The pattern itself works with any broker. The SQL and the transaction boundary are what matter. I'll cover the SQS and RabbitMQ differences after the steps.

How to prevent duplicate messages in Kafka: step by step

To prevent duplicate messages in a Kafka consumer, turn off auto-commit, give each event a stable ID, insert that ID into a dedup table inside the same database transaction as the side effect, and commit the offset only after the transaction succeeds. The idempotent producer handles duplicate writes; this handles duplicate reads.

Step 1: Give every event a stable ID

The ID must be created once, when the event is born, and never again. If you generate a fresh UUID on every retry, every retry looks like a new message and your dedup table catches nothing.

import json
import uuid
from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "enable.idempotence": True,  # already the default, set it so nobody turns it off by accident
})

def publish_order_placed(order_id: str, amount_cents: int) -> None:
    event_id = str(uuid.uuid4())
    producer.produce(
        "orders",
        key=order_id,
        value=json.dumps({"order_id": order_id, "amount_cents": amount_cents}),
        headers=[("event_id", event_id.encode())],
    )
    producer.flush()

Two details here. The key=order_id matters for ordering, which I'll come back to. And enable.idempotence is true by default in the current Kafka producer configs. That change was proposed in KIP-679, which also moved acks from 1 to all.

The Kafka docs explain what the idempotent producer does: the broker assigns each producer an ID and uses a sequence number sent with every message to discard duplicate sends. That stops a network retry from writing the same record twice. It does not stop your consumer from reading a record twice.

Step 2: Create the dedup table

CREATE TABLE processed_messages (
    consumer_name TEXT        NOT NULL,
    message_id    TEXT        NOT NULL,
    processed_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, message_id)
);

CREATE TABLE payments (
    order_id     TEXT PRIMARY KEY,
    amount_cents INTEGER NOT NULL,
    charged_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

The consumer_name column is there because two different services can legitimately process the same event. Your payment service and your email service both need to handle OrderPlaced once each, not once in total. This mirrors the (subscriberId, messageID) key in the microservices.io pattern.

Step 3: Turn off auto-commit

from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "payments",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["orders"])

According to the Kafka consumer configs, enable.auto.commit defaults to true, and the offset is "periodically committed in the background," every 5,000 ms by default. That timer has no idea whether your handler finished. You lose control of the one moment that decides between "lost" and "duplicated". Turn it off and commit by hand.

Step 4: Process inside one transaction, then commit the offset

import json
import psycopg

CONSUMER_NAME = "payments"
db = psycopg.connect("postgresql://localhost/shop", autocommit=True)

def handle(msg) -> None:
    headers = dict(msg.headers() or [])
    message_id = headers["event_id"].decode()
    event = json.loads(msg.value())

    with db.transaction():
        cur = db.execute(
            """
            INSERT INTO processed_messages (consumer_name, message_id)
            VALUES (%s, %s)
            ON CONFLICT DO NOTHING
            """,
            (CONSUMER_NAME, message_id),
        )
        if cur.rowcount == 0:
            return  # seen it before, skip

        db.execute(
            "INSERT INTO payments (order_id, amount_cents) VALUES (%s, %s)",
            (event["order_id"], event["amount_cents"]),
        )

while True:
    msg = consumer.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print(msg.error())
        continue
    handle(msg)
    consumer.commit(message=msg, asynchronous=False)

Walk through the crash cases:

1. Crash before the DB transaction commits. Nothing was written. Kafka redelivers, and you process it for the first time.
2. Crash after the DB commit, before the offset commit. Kafka redelivers. The insert into processed_messages conflicts, rowcount is 0, and you skip. You commit the offset and move on.
3. Crash after both commits. Kafka never redelivers. Nothing to do.

Every path ends with the payment recorded once. That is the whole trick: the dedup record and the side effect live or die together.

Step 5: Add the outbox pattern on the producer side

Step 1 has a hole. If your order service writes the order to its database and then calls producer.produce(), a crash between the two means the order exists but the event never ships. Publishing first has the opposite problem.

What is the outbox pattern?

The transactional outbox pattern saves each outgoing event in an outbox table in the same database transaction as the business change, and a separate relay forwards those rows to the broker. You never get a saved order without its event, or an event without its order. The relay can still publish twice, so consumers stay idempotent.

Here is a minimal version on PostgreSQL, continuing Step 5.

CREATE TABLE outbox (
    id         UUID PRIMARY KEY,
    topic      TEXT        NOT NULL,
    msg_key    TEXT        NOT NULL,
    payload    JSONB       NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    sent_at    TIMESTAMPTZ
);

def place_order(order_id: str, amount_cents: int) -> None:
    with db.transaction():
        db.execute(
            "INSERT INTO orders (id, amount_cents) VALUES (%s, %s)",
            (order_id, amount_cents),
        )
        db.execute(
            "INSERT INTO outbox (id, topic, msg_key, payload) VALUES (%s, %s, %s, %s)",
            (str(uuid.uuid4()), "orders", order_id,
             json.dumps({"order_id": order_id, "amount_cents": amount_cents})),
        )

def relay_once() -> None:
    with db.transaction():
        rows = db.execute(
            """
            SELECT id, topic, msg_key, payload FROM outbox
            WHERE sent_at IS NULL
            ORDER BY created_at
            LIMIT 100
            FOR UPDATE SKIP LOCKED
            """
        ).fetchall()
        for row_id, topic, key, payload in rows:
            producer.produce(topic, key=key, value=json.dumps(payload),
                             headers=[("event_id", str(row_id).encode())])
        producer.flush()
        for row_id, *_ in rows:
            db.execute("UPDATE outbox SET sent_at = now() WHERE id = %s", (row_id,))

Notice that the outbox row's primary key becomes the event_id. The ID is created once, inside the business transaction, which is exactly what Step 1 asked for.

This version is what the pattern calls a "polling publisher". The other option is transaction log tailing, where the relay reads the database's change log instead of polling a table.

Now the honest part. microservices.io states that the relay can publish a message more than once, for example if it crashes after publishing but before recording that it did. That is why the pattern page says "a message consumer must be idempotent." The outbox guarantees you never lose an event. Step 4 guarantees you never apply one twice. You need both.

Step 6: Prove it with a replay test

Don't trust it until you've watched it skip a duplicate.

def test_duplicate_is_skipped():
    class FakeMsg:
        def headers(self): return [("event_id", b"evt-123")]
        def value(self): return b'{"order_id": "o-1", "amount_cents": 5000}'

    handle(FakeMsg())
    handle(FakeMsg())

    count = db.execute("SELECT count(*) FROM payments WHERE order_id = 'o-1'").fetchone()[0]
    assert count == 1

Run it, then kill your real consumer with kill -9 mid-batch and restart it. Check the payments table. One row per order.

How to design an idempotency key

A good idempotency key is unique per logical operation, created once when the operation starts, reused on every retry, and contains no sensitive data. Use a random UUID when the operation has no natural identity, or a deterministic key like order-42:charge when it does. Store it behind a unique constraint.

The two public references agree on the basics. Stripe's API docs suggest V4 UUIDs or another random string with enough entropy, allow keys up to 255 characters, and warn against using emails or personal identifiers. The IETF HTTPAPI working group's Idempotency-Key header draft also recommends a UUID or similar random identifier, and says a key must not be reused with a different payload.

Key type	Example	Use it when	Watch out for
Random UUID, created at origin	uuid4() at event creation	The event has no natural identity	Must be stored and reused on retry, never regenerated
Deterministic business key	order-42:charge	One action per entity is the rule	Two legitimate charges on one order would collide
Content hash	SHA-256 of the body	You can't change the producer	Two genuinely separate but identical events look like one

Stripe adds two details worth copying. It compares incoming parameters against the original request and errors if they differ, which catches a reused key with a new payload. And it may prune keys once they are at least 24 hours old. Your dedup table needs a retention rule too: keep IDs for longer than the longest time a message could realistically be redelivered, then delete old rows in a scheduled job.

How to guarantee message order in RabbitMQ

RabbitMQ keeps messages in publication order within a single queue, even after requeues, but multiple consumers on one queue can still see them out of order. To guarantee processing order, use one queue with Single Active Consumer, enabled with the x-single-active-consumer argument, and process deliveries one at a time.

According to the RabbitMQ semantics docs, AMQP 0-9-1 guarantees that messages published on one channel, through one exchange and one queue, to one outgoing channel arrive in the order sent. Since release 2.7.0, the queue holds messages in publication order even with requeueing. The same page warns that with several subscribers, individual consumers may still observe messages out of order.

Single Active Consumer is the fix. Declare the queue with x-single-active-consumer set to true and only one consumer receives messages at a time. If it disconnects, another registered consumer takes over. It can only be set when you declare the queue, not through a policy.

channel.queue_declare(
    queue="orders",
    durable=True,
    arguments={"x-single-active-consumer": True},
)

The consumer docs add the last piece: applications that depend on strictly sequential processing need a concurrency factor of one, or their own synchronization.

Ordering and deduplication are separate problems. A RabbitMQ consumer that acks after processing still gets redeliveries. The Reliability Guide says the redelivered flag is a hint that a message may have been seen before, and if the flag is not set, the message is guaranteed to be new. Use the flag to log, not to decide. The dedup table from Step 2 is still your source of truth.

In Kafka, ordering comes from the key. The Kafka introduction states that events with the same key go to the same partition, and a consumer of a partition reads its events in exactly the order they were written. That is why Step 1 uses key=order_id: every event for one order lands in one partition, in order.

When does exactly-once really apply?

Exactly-once applies inside Kafka: a Kafka Streams app or a transactional producer that reads from Kafka, processes, and writes back to Kafka, with consumers using isolation.level=read_committed. Once your consumer touches a database, an HTTP API or an email server, Kafka's guarantee ends and you need idempotency.

Kafka has supported transactions since version 0.11.0.0. A transactional producer writes to several partitions atomically, and a consumer at the read_committed isolation level only returns messages from committed transactions. The default, read_uncommitted, can show messages from aborted ones.

The cost is real but small. In Confluent's benchmark, written by Neha Narkhede and Guozhang Wang, a transactional producer with 1 KB messages and 100 ms transactions lost about 3% throughput compared with an at-least-once, in-order producer. Kafka Streams with a 100 ms commit interval lost 15% to 30%, depending on message size.

The same Confluent article is clear about the limit: side effects such as remote calls and external store updates "would not be guaranteed exactly once." The Kafka design docs give the alternative for external systems: store the consumer's offset in the same place as the output, so both update or neither does. The dedup table in Step 4 is a close cousin of that idea.

Common mistakes

These are the failure modes that bite most often, and every one of them passes a happy-path test.

1. Generating the ID at the consumer. A UUID created on receipt is new every time. The ID must travel with the message from the producer.
2. Dedup check and side effect in separate transactions. Check, then process, then insert the ID: a crash in the middle leaves you with the effect and no record. One transaction, always.
3. Committing the offset before processing. The Kafka docs note this is how you get at-most-once. You stop seeing duplicates and start losing messages.
4. Calling an external API inside the DB transaction. The transaction can roll back; the HTTP call cannot. If the external API accepts an idempotency key, as Stripe's does, pass your message ID to it.
5. Trusting the redelivered flag or the idempotent producer as your dedup. Both are helpful signals. Neither covers a consumer crash after a side effect.
6. An SQS visibility timeout shorter than your handler. The default is 30 seconds. If processing can take longer, extend it with ChangeMessageVisibility, up to the 12-hour maximum.
7. Never cleaning the dedup table. It grows forever. Add a retention window and a cleanup job on day one.

Next steps

• Add the dedup table to the one consumer in your system that moves money or sends email. Start there, not everywhere.
• Move every producer that writes to a database and a broker in the same request onto the outbox pattern.
• Audit your SQS visibility timeouts against your slowest handler, and check your Kafka consumers for enable.auto.commit.
• Reserve Kafka transactions for flows that stay inside Kafka. Everything that leaves Kafka gets an idempotency key.

FAQ

Does enable.idempotence prevent duplicate messages in a Kafka consumer?

No. The idempotent producer stops duplicate writes to the log when the producer retries. It does nothing when your consumer crashes after processing and before committing its offset. For that you need an idempotent consumer with a dedup table, or writes that are naturally idempotent.

Is Kafka exactly-once delivery real?

Yes, within Kafka. Transactions plus read_committed consumers give exactly-once for read-process-write flows that stay inside Kafka, and Kafka Streams uses this. Once a side effect leaves Kafka, such as a database write or an email, you need idempotency or to store the offset with the output.

Can the RabbitMQ redelivered flag replace a dedup table?

No. If the flag is not set, RabbitMQ guarantees the message is new, but a set flag only means it might have been seen before. Publishers can also create duplicates when a confirm is lost, and those arrive without the flag. The Reliability Guide recommends designing consumers to be idempotent.

How long should I keep message IDs in the dedup table?

Longer than the longest realistic redelivery window for your system: consumer lag, retries, and any manual replays. Stripe prunes idempotency keys after at least 24 hours, which is a sensible reference point for request retries. If you replay topics from days ago, keep IDs for at least that long.

Pick the one consumer that moves money in your system and add the dedup table to it this week. If you're preparing to explain this in an interview, read why most developers fail system design interviews, and see how WhatsApp handles billions of messages for messaging at scale. When you need to inspect a payload, the free tools on Art of Code are a click away.

Written by Adithya Guttha, Founder of Art of Code.