Messaging & Event Streaming

Kafka Producers and Consumers in Java: Offsets, Idempotence and Transactions

Intermediate7 min readUpdated

The Kafka client API is small, but the settings behind it decide whether you lose records, duplicate them or stall a whole consumer group. This guide explains the producer's batching and delivery pipeline and its idempotent mode, then the consumer's poll loop, offset commit strategies and rebalance listeners, and finishes with transactions for exactly-once processing and the Spring Kafka equivalents — with complete Java examples for kafka-clients 3.x.

What you will learn

  • Configure a KafkaProducer for throughput or durability and explain what each setting changes
  • Send records with callbacks and understand idempotence, ordering and delivery.timeout.ms
  • Write a consumer poll loop that commits offsets safely and survives rebalances
  • Choose between auto-commit, commitSync and commitAsync and know the duplicate window of each
  • Use transactions for read-process-write pipelines with exactly-once semantics
  • Map the raw client API onto KafkaTemplate and @KafkaListener in Spring Kafka

The Kafka Java client hides a lot of machinery behind two classes. KafkaProducer.send looks like a method call but is the front of an asynchronous batching pipeline with retries, and KafkaConsumer.poll looks like "give me messages" but also drives heartbeats, group membership and offset management. Getting reliable behaviour means understanding what those two calls do and choosing a handful of settings deliberately. This guide uses org.apache.kafka:kafka-clients 3.x; the API is stable across 3.x and 4.x.

The producer pipeline

When you call send, the record is serialised, assigned to a partition, and appended to an in-memory batch for that partition. A background I/O thread ships full batches — or batches older than linger.ms — to the partition leaders, waits for acknowledgements according to acks, retries failures, and finally completes the Future and invokes your callback. Nothing has left the process when send returns.

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092,kafka-2:9092,kafka-3:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.CLIENT_ID_CONFIG, "order-service");

// Durability: wait for the in-sync replicas, never reorder, never duplicate on retry
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000);   // total budget for retries

// Throughput: batch for up to 20 ms, compress batches
props.put(ProducerConfig.LINGER_MS_CONFIG, 20);
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");

try (Producer<String, String> producer = new KafkaProducer<>(props)) {
  ProducerRecord<String, String> record =
      new ProducerRecord<>("orders", "customer-4711", "{\"orderId\":\"o-9911\",\"total\":149.90}");
  record.headers().add("schemaVersion", "2".getBytes(StandardCharsets.UTF_8));

  producer.send(record, (metadata, exception) -> {
    if (exception != null) {
      log.error("failed to send order o-9911", exception);         // after all retries were exhausted
    } else {
      log.debug("o-9911 → {}-{} @ {}", metadata.topic(), metadata.partition(), metadata.offset());
    }
  });
  // close() flushes pending batches and waits for their callbacks
}

The settings that matter

SettingDefault (3.x)What it does
acksallHow many replicas must persist the record before it counts as sent
enable.idempotencetrueBroker de-duplicates retried batches using a producer id and sequence numbers; guarantees per-partition ordering with up to 5 in-flight requests
retriesInteger.MAX_VALUERetry transient errors (leader election, network) until delivery.timeout.ms expires
delivery.timeout.ms120 000Upper bound on the time from send to success or failure, including retries
linger.ms0How long to wait for more records before shipping a non-full batch; a few milliseconds multiplies throughput
batch.size16 384 bytesMaximum bytes per partition batch
compression.typenonelz4 or zstd shrink batches at low CPU cost; compression applies per batch, so it pairs with linger.ms
buffer.memory32 MBTotal memory for unsent batches; when full, send blocks for max.block.ms then throws
max.request.size1 MBLargest request; the broker's message.max.bytes must agree

Idempotence is the important one. Without it, a batch that was written by the leader but whose acknowledgement was lost is retried and written twice, and a retried earlier batch can land behind a later one. With it, the broker recognises the retry by its sequence number and keeps exactly one copy in order. It costs nothing measurable, and it is enabled by default with acks=all — but any explicit acks=1 or acks=0 silently disables it, which is a common reason for duplicates after an upgrade.

send also throws synchronously for a few conditions: serialisation failure, a record too large for max.request.size, and a full buffer after max.block.ms. Handle both the exception and the callback.

The consumer poll loop

A consumer subscribes to topics, then calls poll in a loop. Each poll returns a batch of records from the partitions assigned to this consumer and, as a side effect, sends heartbeats, joins or rejoins the group, and — with auto-commit — commits offsets. That side effect is why the loop must keep polling even when there is nothing to process.

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092,kafka-2:9092,kafka-3:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "billing");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");  // first start: read from the beginning
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);     // we commit after processing
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 200);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300_000);

try (Consumer<String, String> consumer = new KafkaConsumer<>(props)) {
  consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
    @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
      consumer.commitSync();          // commit what we finished before losing the partitions
    }
    @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { }
  });

  while (running.get()) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> record : records) {
      billing.process(record.key(), record.value());       // must be idempotent: redelivery is possible
    }
    consumer.commitSync();            // after the whole batch: at-least-once
  }
} catch (WakeupException ignored) {
  // consumer.wakeup() was called from a shutdown hook
}

auto.offset.reset only applies when the group has no committed offset for a partition: earliest processes history, latest (the default) starts from new records only. Once the group has committed, the setting is irrelevant, which surprises people who change it and see no effect.

max.poll.records bounds how much work one iteration holds, and max.poll.interval.ms bounds how long an iteration may take. If processing 200 records can take longer than five minutes in the worst case, lower the first or raise the second, or move the work to another thread and call pause on the partitions while it runs so that poll can keep the membership alive.

Stop a consumer from another thread with consumer.wakeup(), which makes the blocked poll throw WakeupException; then close the consumer on its own thread. Closing commits nothing by itself unless auto-commit is on, so the shutdown path should commit explicitly.

Committing offsets

The offset you commit is the position of the next record to read. Where in the loop you commit decides the size of the duplicate or loss window after a crash.

StrategyHowAfter a crash
Auto-commit (enable.auto.commit=true, every auto.commit.interval.ms)Committed on the next poll, covering everything the previous poll returnedRecords returned but not yet processed can be lost if you hand them to another thread; otherwise duplicates up to one interval
commitSync() after processing the batchBlocks until the coordinator acknowledges; retries on transient errorsDuplicates limited to the current batch
commitAsync() after processingNon-blocking; no retries, because a retry could overwrite a newer commitSame as above, slightly larger window when commits fail
Per-record commitSync(Map)Commit a specific offset after each recordSmallest window, highest overhead

A common pattern combines the two commit forms: commitAsync in the loop for throughput, and commitSync in the finally block and in onPartitionsRevoked for correctness. Whatever you choose, the consumer must tolerate seeing a record twice — a crash between processing and committing cannot be prevented — so processing should be idempotent, typically by recording (topic, partition, offset) or a business key alongside the result.

If offsets are better stored with the data — in the same database transaction as the row the record produced — disable auto-commit, write the offset into the database, and on startup read it back and seek each assigned partition to it from onPartitionsAssigned. That gives exactly-once between Kafka and that database without Kafka transactions.

Transactions and exactly-once

For pipelines that read from Kafka, process, and write back to Kafka, transactions make the output records and the input offset commit atomic: either the outputs are visible and the offsets are committed, or neither. Together with idempotence this is Kafka's exactly-once semantics (EOS).

Properties pp = producerProps();
pp.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "billing-enricher-" + instanceId);  // stable per instance
Producer<String, String> producer = new KafkaProducer<>(pp);
producer.initTransactions();

Properties cp = consumerProps();
cp.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");   // never see aborted records
cp.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
Consumer<String, String> consumer = new KafkaConsumer<>(cp);
consumer.subscribe(List.of("orders"));

while (running.get()) {
  ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
  if (records.isEmpty()) continue;
  producer.beginTransaction();
  try {
    for (ConsumerRecord<String, String> r : records) {
      producer.send(new ProducerRecord<>("invoices", r.key(), enrich(r.value())));
    }
    // Offsets are committed as part of the transaction, not by the consumer
    producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());
    producer.commitTransaction();
  } catch (ProducerFencedException | OutOfOrderSequenceException e) {
    throw e;                              // another instance took over this transactional id; stop
  } catch (KafkaException e) {
    producer.abortTransaction();          // inputs will be re-polled and reprocessed
  }
}

transactional.id identifies the producer across restarts: a new instance with the same id fences the old one, so a zombie cannot commit stale work. Downstream consumers only see the outputs once read_committed is set; with the default read_uncommitted they would also see records from aborted transactions. Transactions cost latency — a commit is a round trip to the transaction coordinator plus markers in every partition written — and are best used for batches of records, not one at a time.

Kafka Streams enables all of this with a single setting (processing.guarantee=exactly_once_v2), which is usually the easier route when the whole pipeline is Kafka-to-Kafka.

Serialisation

The client ships serialisers for strings, integers, longs, byte arrays and a few more. Real payloads use JSON with a schema, or Avro or Protocol Buffers through a schema registry, whose serialisers prepend a schema identifier to each record so that consumers can evolve independently. Whatever the format, deserialisation failures on the consumer side are fatal to the poll loop unless handled: the ErrorHandlingDeserializer in Spring Kafka, or a wrapper that returns a sentinel, lets the loop skip a corrupt record and continue rather than crash on it forever.

Spring Kafka

Spring for Apache Kafka wraps the same client. KafkaTemplate.send returns a CompletableFuture over the same producer; @KafkaListener runs a poll loop per container thread, converts records, and commits according to an AckMode (BATCH after each poll by default, MANUAL when you call Acknowledgment.acknowledge() yourself, RECORD after every record).

@Component
class OrderListener {

  @KafkaListener(topics = "orders", groupId = "billing", concurrency = "3")
  public void onOrder(ConsumerRecord<String, String> record, Acknowledgment ack) {
    billing.process(record.key(), record.value());
    ack.acknowledge();                                  // AckMode.MANUAL: commit after this record's batch
  }
}

concurrency = "3" creates three consumer threads in the group, each with its own KafkaConsumer, so it should not exceed the partition count. Boot's spring.kafka.* properties configure serialisers, acks, group id and the listener's acknowledgement mode, and DefaultErrorHandler with a DeadLetterPublishingRecoverer gives you the retry-then-dead-letter-topic behaviour that raw Kafka lacks:

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
  var recoverer = new DeadLetterPublishingRecoverer(template);            // → orders.DLT by default
  return new DefaultErrorHandler(recoverer, new FixedBackOff(1_000L, 3)); // 3 retries, 1 s apart
}

Because a failed record blocks its partition while it is retried, keep the backoff short and move anything that needs long waits to a separate retry topic; Spring Kafka's @RetryableTopic automates that pattern.

Common mistakes

  • acks=1 or acks=0 "for performance". It disables idempotence and accepts loss on leader failure. Batching and compression give far more throughput without the loss.
  • Fire-and-forget without a callback. send returns before anything happens; a failed record vanishes unless the callback or the Future is checked.
  • Processing in another thread with auto-commit on. The next poll commits records that thread has not finished. Commit manually after processing, or pause partitions.
  • Consumers without an idempotent side effect. At-least-once means duplicates will happen — after a rebalance, a crash, or a retry. De-duplicate on a key or an offset.
  • Blocking inside the poll loop for longer than max.poll.interval.ms. The consumer is evicted, the group rebalances, and the eventual commit fails. Bound the work per poll.
  • Using commitAsync everywhere. Its commits are not retried, so a transient failure at shutdown loses progress. Finish with commitSync.
  • Sharing one KafkaConsumer between threads. The client detects it and throws ConcurrentModificationException; use one consumer per thread.

Key takeaways

  • send appends to a batch; an I/O thread ships, retries and acknowledges. Keep acks=all and idempotence on, tune linger.ms, batch.size and compression for throughput, and always check the callback.
  • poll is the heartbeat of the consumer: keep calling it, bound the work per call with max.poll.records, and stay under max.poll.interval.ms.
  • The committed offset is the next record to read; commit after processing for at-least-once and make processing idempotent, because duplicates after a crash or rebalance cannot be avoided.
  • Use commitAsync for speed in the loop and commitSync on revocation and shutdown; store offsets in the target database when you need exactly-once into a non-Kafka system.
  • Transactions with transactional.id, sendOffsetsToTransaction and read_committed consumers give exactly-once for Kafka-to-Kafka pipelines; Kafka Streams packages the same thing.
  • Spring Kafka's KafkaTemplate, @KafkaListener, DefaultErrorHandler and dead-letter recoverer wrap the client without changing any of these rules.

Sources and further reading

Frequently asked questions

Is KafkaProducer thread-safe? Is KafkaConsumer?
KafkaProducer is thread-safe and one instance per application is the recommended pattern; sharing it across threads is faster than creating several. KafkaConsumer is not thread-safe: all calls must come from one thread, and the usual design is one consumer per thread, with a wakeup() call from another thread to stop it.
Why does my producer send records out of order?
With retries enabled and several requests in flight, a failed batch can be retried after a later batch succeeded. Enable idempotence (the default since Kafka 3.0), which keeps ordering per partition with up to five in-flight requests, or set max.in.flight.requests.per.connection to 1 if you must run without it.
What happens if my consumer takes too long to process a batch?
If the time between two poll() calls exceeds max.poll.interval.ms (five minutes by default), the consumer is considered failed, leaves the group and its partitions are reassigned. Its next commit then fails with a CommitFailedException. Reduce max.poll.records, process faster, or pause the partitions while work runs in the background and keep polling.
Does exactly-once work when my consumer writes to a database?
Kafka transactions cover writes to Kafka topics and offset commits atomically. A database write is outside that transaction. For Kafka-to-database pipelines the reliable approaches are an idempotent write keyed by topic, partition and offset, or storing the offset in the database in the same transaction as the data and seeking to it on startup.