Messaging & Event Streaming

Messaging Patterns and Delivery Guarantees: Idempotency, Outbox, Retries and Ordering

Advanced12 min readUpdated

Every broker promises reliability; none of them can make your application reliable on its own. This guide explains what at-most-once, at-least-once and exactly-once really mean and where duplicates and losses come from, then works through the patterns that close the gaps — idempotent consumers, the transactional outbox, retry with backoff and dead-letter queues, ordering per key, request-reply, competing consumers, claim check and schema versioning — and shows how each maps onto ActiveMQ, RabbitMQ and Kafka.

What you will learn

  • Explain the three delivery guarantees and identify exactly where a duplicate or a loss can occur in a pipeline
  • Design an idempotent consumer with a de-duplication store and a sensible retention window
  • Implement the transactional outbox so that database writes and published events cannot diverge
  • Build retry, backoff and dead-letter handling that does not block healthy messages
  • Preserve ordering where it matters using keys, message groups, partitions and single active consumers
  • Apply request-reply, competing consumers, claim check and schema evolution correctly on each broker

Brokers advertise durability, acknowledgements and transactions, and it is easy to conclude that using one makes a system reliable. It does not. Reliability is a property of the whole path — producer, broker, consumer and every side effect the consumer causes — and the broker only controls the middle. The patterns in this guide are the ones that make the ends reliable too. They are broker-independent; the last section shows how each maps onto ActiveMQ Classic, RabbitMQ and Kafka, which the earlier guides in this subject cover in detail.

What the three guarantees actually mean

Delivery guarantees describe what happens to a message when something fails between the moment a producer sends it and the moment a consumer finishes with it.

At-most-once. The producer sends without waiting for confirmation, or the consumer acknowledges before processing. A failure anywhere loses the message; nothing is ever duplicated. It is the right choice for data that is superseded quickly — metrics samples, presence updates, cache invalidations — where a lost message is corrected by the next one.

At-least-once. The producer waits for the broker's confirmation and retries if it does not arrive; the consumer acknowledges only after processing. Nothing is lost, but every retry can create a duplicate: the producer's first send may have succeeded before the confirmation was lost, and the consumer may have completed its work before crashing short of the acknowledgement. This is what nearly all production systems run.

Exactly-once. Each message affects the system precisely once. No broker can guarantee this for an arbitrary consumer, because the consumer's side effects — an HTTP call, a row in another database — are outside the broker's control. What brokers can offer is exactly-once within a boundary: Kafka transactions make a read-process-write cycle between Kafka topics atomic, and a JMS transacted session makes consume-plus-produce on one broker atomic. Outside the boundary, exactly-once is something you build with at-least-once delivery and idempotent processing.

Where duplicates and losses come from is worth memorising, because each pattern below targets one of these points:

Point of failureSymptom without a patternPattern
Producer crashes after the database commit, before the publishLost event, inconsistent systemsTransactional outbox
Producer's confirmation is lost, producer retriesDuplicate messageIdempotent producer (Kafka), broker de-duplication, or idempotent consumer
Broker crashes before persistingLost messagePersistent delivery, confirms, replication
Consumer crashes after side effect, before acknowledgementDuplicate side effectIdempotent consumer
Consumer cannot process a messageBlocked queue or infinite loopRetry with backoff, dead-letter queue
Parallel consumersReorderingOrdering per key

Idempotent consumer

An idempotent consumer produces the same end state whether it processes a message once or several times. It is the single most valuable pattern in messaging because it makes at-least-once delivery safe, and at-least-once is what you have.

Some operations are naturally idempotent: setting a status to SHIPPED, upserting a row keyed by order id, deleting a record. Others are not — incrementing a counter, charging a card, sending an email — and need a de-duplication store: a table or key-value entry recording which message identifiers have been processed, checked and written in the same transaction as the side effect.

@Transactional
public void handle(OrderPaid event, String messageId) {
  // INSERT ... ON CONFLICT DO NOTHING; returns 0 rows when the id was already recorded
  if (processedMessages.insertIfAbsent(messageId, Instant.now()) == 0) {
    return;                                 // duplicate: already handled, acknowledge and move on
  }
  ledger.credit(event.merchantId(), event.amount());
  // commit writes the ledger entry and the de-duplication row atomically
}

Three details decide whether this works. The identifier must be assigned by the producer — a message id it generates once, or a business key such as the order id plus the event type — because a broker-assigned id changes when the producer retries. The check and the side effect must be in one transaction, or a crash between them recreates the problem. And the store must expire entries after a window longer than any possible redelivery, or it grows forever. When the side effect is a call to another system, pass the same identifier along as an idempotency key so that the downstream system can de-duplicate too; payment providers and most well-designed APIs accept one.

Transactional outbox

A service that saves an order and publishes an OrderPlaced event has a dual-write problem: the database and the broker are two systems, and there is no transaction that spans both. Publishing after the commit risks a saved order with no event; publishing before it risks an event for an order that was rolled back.

The transactional outbox removes the second system from the transaction. The service inserts the event into an outbox table in the same transaction as the order. A separate relay reads new outbox rows and publishes them, marking each as sent. If the relay crashes after publishing and before marking, the event is published twice — which is at-least-once, and the idempotent consumer above absorbs it.

CREATE TABLE outbox (
  id            UUID PRIMARY KEY,           -- becomes the message id
  aggregate_id  TEXT NOT NULL,              -- becomes the routing key or partition key
  event_type    TEXT NOT NULL,
  payload       JSONB NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at  TIMESTAMPTZ
);
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

The relay can be a scheduled poller (SELECT ... WHERE published_at IS NULL ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED, publish, update) or a change data capture tool such as Debezium that tails the database's transaction log and publishes every outbox insert to Kafka with no polling and strict ordering. Polling is simpler and fine at moderate volume; CDC scales further and adds no load to the tables. Either way, delete or archive published rows so that the table stays small.

The outbox also gives you event ordering per aggregate for free: rows are published in insertion order, and using aggregate_id as the message group or partition key keeps that order on the consumer side.

Retries, backoff and dead letters

A consumer that fails on a message has three honest options: retry, skip, or park. The wrong option — retry immediately and forever — is the default behaviour of many listener containers, and it turns one poison message into a queue that never drains and a dependency that is hammered until it falls over.

A sound policy has four parts:

  1. Classify the failure. A timeout or a 503 is transient and worth retrying. A validation error or a 404 for a referenced entity is permanent; retrying it wastes attempts.
  2. Back off exponentially with jitter. One second, two, four, eight, with a random fraction added so that a hundred consumers failing together do not retry in lockstep. Cap the delay and the number of attempts.
  3. Retry without blocking the queue. In-memory retry holds the consumer thread and the messages behind it. For delays beyond a few seconds, republish the message to a retry destination with a delay — a TTL queue that dead-letters back to the work queue in RabbitMQ, a scheduled delivery in ActiveMQ, a retry topic consumed after a wait in Kafka — and acknowledge the original.
  4. Dead-letter what remains. After the last attempt, move the message to a dead-letter queue (DLQ) with the exception, stack trace, attempt count and original destination attached as headers. A DLQ without a reader is a black hole: alert on its depth, and keep a tool to inspect and replay its contents.

Skipping deserves a mention: for streams where one bad record must not stop the pipeline — analytics, logs — logging the record and moving on is a legitimate policy, as long as it is a decision and not an accident.

Ordering

Brokers preserve order in exactly one place: a single queue or partition read by a single consumer. Everything that adds throughput — competing consumers, multiple partitions, redelivery after a failure — weakens it. The workable position is ordering per key: all messages about one entity arrive in order, while messages about different entities may interleave.

BrokerMechanismConsumer side
ActiveMQ ClassicJMSXGroupID message groups: all messages with a group id go to one consumer while it is aliveNothing special; the broker pins the group
RabbitMQOne queue per key range (consistent-hash exchange plugin) or a single queue with x-single-active-consumerOne active consumer per queue
KafkaPartition by key; the default partitioner does thisOne consumer per partition within a group

Two subtleties remain even with per-key ordering. A retried message can overtake later messages for the same key unless retries are also per key — the safest fix is to stop the key's processing until the retry succeeds, which Kafka does naturally by blocking the partition. And an event's meaning should not depend on strict order where you can avoid it: carrying a version number or a timestamp in every event lets a consumer ignore a stale update that arrives late, which is cheaper than perfect ordering.

Interaction patterns

Competing consumers

Several consumers reading one queue share its messages, and adding a consumer adds throughput until the producer or a downstream dependency becomes the limit. It is the default scaling model for queues and, with consumer groups, for Kafka. Keep the prefetch or max.poll.records modest so that work is shared evenly and a crash redelivers little.

Publish-subscribe

One event, many independent readers, each with its own position and its own failures. Implement it with topics and durable subscriptions in JMS, a fanout or topic exchange feeding one queue per subscriber in RabbitMQ, and independent consumer groups in Kafka. Each subscriber must own its own retries and dead letters; a slow subscriber must never be able to block the others.

Request-reply

A message that expects an answer needs a correlation identifier that the reply carries back and a reply address the responder sends to. Give every request a timeout and decide what the requester does when it expires — a reply arriving after the timeout must be recognised as stale and discarded. Because request-reply reintroduces temporal coupling, reserve it for cases where the caller truly cannot continue without the answer.

Claim check

Brokers are built for small messages; a 20 MB payload slows every hop and can exceed frame limits. The claim check pattern stores the payload in object storage or a database and sends a message containing only a reference and the metadata a router needs. The consumer fetches the payload when it processes the message. Expire stored payloads with the same discipline as de-duplication records.

Content-based routing and message translation

When a message must go to different consumers based on its attributes, route on headers or routing keys at the broker (a topic exchange, a selector, a topic per type), not by having every consumer receive everything and discard most of it. When two systems disagree on format, put the translation in one place — a dedicated consumer that republishes — rather than in every consumer.

Schema evolution

Messages outlive the code that produced them. A queue can hold yesterday's format when today's consumer deploys, and a Kafka topic holds a week of history. Every message therefore needs a schema version, and every change must be classified as compatible or breaking:

  • Compatible: adding an optional field with a default, widening a numeric type, adding a new event type on a new routing key. Consumers ignore unknown fields.
  • Breaking: renaming or removing a field, changing a type or its meaning, changing the key. These require a new topic or routing key, or a translation step, and a period where both versions are produced or consumed.

A schema registry (Avro, Protobuf or JSON Schema) enforces compatibility rules at publish time and is worth adopting as soon as more than one team consumes a topic. Without one, a schemaVersion header and a consumer that validates before processing are the minimum; use the JSON formatter to inspect payloads and the Base64 decoder when a tool shows the body encoded.

Mapping patterns to brokers

PatternActiveMQ ClassicRabbitMQKafka
At-least-oncePersistent delivery, AUTO/CLIENT ack or transacted sessionDurable quorum queue, persistent message, confirms, manual ackacks=all, idempotent producer, commit after processing
Broker-side exactly-once boundaryTransacted session (consume + produce on one broker), XA through an application serverNone; use idempotent consumersTransactions with read_committed consumers, Kafka Streams EOS
Producer de-duplicationNone; use an outbox idNone; use an outbox idIdempotent producer (per session)
Delayed retryScheduled delivery (AMQ_SCHEDULED_DELAY)TTL queue with dead-letter exchange back to the work queueRetry topics consumed after a delay
Dead lettersActiveMQ.DLQ or per-destination DLQ, automatic after redelivery policyDead-letter exchange, x-delivery-limit on quorum queuesNot built in; error topic written by the consumer or framework
Ordering per keyJMSXGroupIDSingle active consumer, consistent-hash exchangePartition key
Pub-subTopics, durable and shared subscriptionsFanout/topic exchange with a queue per subscriberConsumer groups
ReplayNoStreams onlyReset offsets
Request-replyTemporary queues, JMSReplyTo, JMSCorrelationIDDirect reply-to, reply_to and correlation_id propertiesReply topic keyed by correlation id; awkward

The table makes one thing visible: Kafka provides the strongest producer-side and broker-side guarantees and the weakest per-message tooling, while the queue brokers provide rich per-message handling and expect you to solve de-duplication yourself. Neither removes the need for idempotent consumers and an outbox.

Common mistakes

  • Trusting "exactly-once" on the box. It applies within the broker's boundary. Side effects outside it need idempotency.
  • Broker-assigned identifiers for de-duplication. They change on producer retry. Generate the id in the producer or derive it from a business key.
  • De-duplication outside the transaction. Checking the store, doing the work, then recording the id leaves a crash window that reproduces the duplicate. One transaction.
  • Publishing after commit without an outbox. Works until the first crash between the two, which is then nearly impossible to reconstruct.
  • Immediate, unbounded retries. Poison messages block queues and retries amplify outages. Back off, cap, dead-letter.
  • Assuming global order. It does not exist beyond one consumer on one queue or partition. Design per key and carry version numbers.
  • Big payloads on the bus. Use a claim check; keep messages to kilobytes.

Key takeaways

  • At-least-once is what you get from any production broker; exactly-once exists only inside a broker's transaction boundary, and at-most-once is a deliberate choice for disposable data.
  • Duplicates come from producer retries and consumer crashes before acknowledgement; the idempotent consumer, with a producer-assigned id checked inside the side effect's transaction, absorbs both.
  • The transactional outbox makes database writes and published events consistent by putting the event in the database and relaying it afterwards.
  • Retry transient failures with capped exponential backoff, park permanent failures in a dead-letter queue with the reason attached, and never let one bad message stop the queue.
  • Order is preserved per queue or partition with one consumer; design for ordering per key with message groups, single active consumers or partition keys, and carry version numbers.
  • Choose interaction patterns deliberately — competing consumers, pub-sub, request-reply with timeouts, claim check for large payloads — and version every message schema from the first release.

Sources and further reading

Frequently asked questions

If my broker supports exactly-once, do I still need idempotent consumers?
Yes, unless every side effect of your consumer is itself inside the broker's transaction. Kafka's exactly-once covers Kafka-to-Kafka processing; the moment a consumer calls a payment API or writes a database row, that effect is outside the transaction and can be repeated after a crash. Idempotency at the consumer is the only mechanism that works regardless of the broker.
Why not just publish the event after the database commit?
Because the process can die between the commit and the publish, leaving a saved order with no event, or the publish can succeed and the commit fail, leaving an event for an order that does not exist. The transactional outbox writes the event into the same database transaction as the data and lets a separate relay publish it, so the two can never disagree.
How long should I keep de-duplication records?
Longer than the longest plausible redelivery. That is bounded by the broker's retention or redelivery window: a few hours covers most queue redeliveries, while Kafka consumers rewinding a week of history need a week. Store the message identifier with a timestamp and expire entries in a background job or with a database TTL.
Should failed messages be retried immediately?
Only once or twice, for network blips. Beyond that, immediate retries hammer a dependency that is already struggling and block the messages behind them. Use exponential backoff with jitter, cap the attempts, and move the message to a dead-letter destination with the failure reason attached so that healthy messages keep flowing.
Message-Oriented Middleware: Queues, Topics and Why Systems Talk AsynchronouslyWhat a message broker does, how queues and topics decouple producers from consumers, which protocols exist, and how to decide when messaging beats an HTTP call.Beginner11 min readKafka Producers and Consumers in Java: Offsets, Idempotence and TransactionsKafka clients in Java: producer batching, acks and idempotence, the consumer poll loop, offset commits, rebalances, exactly-once transactions and Spring Kafka.Intermediate7 min readRabbitMQ with Spring AMQP and Spring Boot: RabbitTemplate, @RabbitListener and RetriesSpring AMQP in Spring Boot 3: declare queues as beans, publish JSON with RabbitTemplate, consume with @RabbitListener, handle confirms, retries, dead letters.Intermediate7 min readArchitectureDistributed Transactions and Consistency: 2PC, Sagas, Outbox and CAPACID versus BASE, what CAP and PACELC really say, why two-phase commit blocks, and how sagas, the outbox and idempotency work.Advanced11 min readArchitectureMicroservice Design Patterns: Resilience, Discovery, Data and ObservabilityA working catalogue of microservice patterns — gateway, discovery, circuit breaker, bulkhead, retry, saga, outbox, CQRS and service mesh.Advanced9 min readArchitectureDomain-Driven Design: Ubiquitous Language, Aggregates and Bounded ContextsHow DDD builds software around a shared model of the business — bounded contexts, aggregates in Java, context maps and hexagonal layering.Advanced9 min read