Messaging & Event Streaming

Apache Kafka Architecture: Topics, Partitions, Replication, Consumer Groups and KRaft

Intermediate11 min readUpdated

Kafka is not a queue with extra features; it is a replicated, partitioned log that consumers read at their own pace. This guide builds that model from the ground up — records, partitions, offsets, segments, retention and compaction, leader and follower replicas, the in-sync replica set, consumer groups and rebalancing, and the KRaft metadata quorum that replaced ZooKeeper — so that producer settings, operations and design decisions later make sense.

What you will learn

  • Explain why Kafka is a log rather than a queue and what that changes for consumers
  • Describe partitions, offsets, segments, retention and compaction and how they interact
  • Explain leader/follower replication, the in-sync replica set, acks and min.insync.replicas
  • Describe consumer groups, offset commits and rebalancing, and their effect on ordering and parallelism
  • Explain the role of the KRaft controller quorum and how it replaced ZooKeeper
  • Decide partition counts, keys and replication factors for a new topic

Apache Kafka started as a way to move activity logs at LinkedIn's scale and grew into the default backbone for event streaming. It is often introduced as "a message queue that scales", which misleads people into using it like one. Kafka is a distributed commit log: an append-only sequence of records, split into partitions spread across brokers, replicated for safety, retained for a configured time, and read by consumers who remember their own position. Every other feature follows from that design. This guide covers Kafka 3.x in KRaft mode and treats ZooKeeper as history.

Records, topics and partitions

A record is a key, a value, a timestamp and optional headers. Key and value are byte arrays to Kafka; serialisation is entirely the client's business. A topic is a named category of records, and it is split into one or more partitions. Each partition is an ordered, immutable sequence: a record appended to partition 3 gets the next offset in partition 3, a 64-bit number that never changes and never repeats within that partition.

Three consequences shape every Kafka design:

  • Ordering exists within a partition and nowhere else. Two records in different partitions have no defined order.
  • Parallelism is bounded by the partition count. A partition is a unit of storage on one broker and a unit of consumption for one consumer in a group.
  • The key decides the partition. The default partitioner hashes the key and takes it modulo the partition count, so all records for customer-4711 land on the same partition and are therefore consumed in order. Records with a null key are spread across partitions in batches.
topic "orders" with 3 partitions

partition 0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 |      ← offsets; new records append on the right
partition 1 | 0 | 1 | 2 | 3 |
partition 2 | 0 | 1 | 2 | 3 | 4 | 5 |

key "customer-4711" → hash → partition 1, always

A record's position is fully identified by (topic, partition, offset), and that triple is what a consumer stores as its progress. There is no per-record acknowledgement, no "unacked" state and no redelivery queue; there is only "the consumer group has read up to here".

Segments, retention and compaction

On disk, a partition is a directory of segment files. The active segment receives appends; when it reaches log.segment.bytes (1 GB by default) or log.roll.ms, it is closed and a new one starts. Each segment has an offset index and a time index beside it so that a consumer asking for "offset 1,234,567" or "the first record after 09:00" is served with a couple of seeks rather than a scan. Writes are sequential appends that go through the operating system's page cache, and reads of recent data are served from that cache; this, more than any clever code, is why Kafka pushes hundreds of megabytes per second per broker on ordinary disks.

Records are removed by retention, not by consumption. Two policies exist per topic:

cleanup.policyRuleUse
delete (default)Drop whole segments older than retention.ms (7 days by default) or beyond retention.bytesEvent streams, logs, anything time-bounded
compactKeep at least the latest record for each key; older records with the same key are removed in the backgroundChangelogs, current state per entity, configuration, the internal offsets topic
compact,deleteBoth: compact, and also drop data older than the retentionChangelogs that should not keep long-dead keys forever

Compaction turns a topic into a key-value snapshot that can be rebuilt by reading it from the start, which is how Kafka Streams state stores and many "materialised view" services recover after a restart. A record with a null value is a tombstone: it marks the key as deleted and is itself removed after delete.retention.ms. Compaction only touches closed segments and runs when the ratio of dirty data justifies it, so a compacted topic always contains some duplicates; readers must expect them.

Retention is what makes the log replayable. A new consumer group can start from the earliest offset and process a week of history; a group that found a bug can rewind to yesterday and reprocess. No queue offers that, and it is the strongest reason to choose Kafka.

Brokers, replication and the in-sync replica set

A broker is one Kafka server process. Each partition has a replication factor — three is the production norm — and its replicas live on that many brokers. One replica is the leader: all produce and, by default, all fetch requests for that partition go to it. The others are followers that continuously fetch from the leader to stay current, exactly like consumers.

The leader tracks which followers are caught up. A follower that has fetched up to the leader's end within replica.lag.time.max.ms (30 seconds by default) is in the in-sync replica set (ISR). The ISR matters for two reasons:

  1. A record is committed when every replica in the ISR has written it. Consumers can only read committed records, up to the high watermark.
  2. If the leader fails, the controller elects a new leader from the ISR, which guarantees that no committed record is lost. With unclean.leader.election.enable=false (the default) an out-of-sync replica is never chosen, so a partition whose ISR is empty becomes unavailable rather than losing data.

The producer's acks setting decides how long it waits:

acksProducer returns whenRisk
0The record has been handed to the networkLoss on any failure; not even a broker error is reported
1The leader has written itLoss if the leader fails before followers fetch it
all (default since 3.0)Every ISR member has written itNone from broker failure — as long as the ISR is larger than one

That last clause is why min.insync.replicas exists. With replication factor 3 and min.insync.replicas=2, the leader rejects writes with acks=all when fewer than two replicas are in sync, trading availability for the guarantee that an acknowledged record exists on at least two brokers. The common production triple is replication factor 3, min.insync.replicas 2, acks=all: one broker can be down for maintenance without blocking producers, and no acknowledged record depends on a single machine.

Since Kafka 2.4, consumers can be told to fetch from the closest replica (replica.selector.class on the broker and client.rack on the consumer), which cuts cross-zone traffic in cloud deployments.

Consumer groups and offsets

A consumer group is a set of consumer instances sharing a group.id. Kafka assigns each partition of the subscribed topics to exactly one member of the group; a group with more members than partitions leaves the extra members idle. Two independent groups reading the same topic each receive every record — that is how Kafka provides both work sharing (within a group) and publish-subscribe (across groups).

Each group's progress is a set of committed offsets, one per partition, stored in the internal __consumer_offsets topic (compacted, keyed by group, topic and partition). A committed offset is "the next record to read", so committing offset 42 means records 0 to 41 are done. Consumers commit periodically or explicitly; on restart or reassignment the new owner of the partition starts from the last committed offset. Everything between the last commit and the crash is read again, which is at-least-once delivery, and why the consumer guide spends so much time on when to commit.

group "billing"     partition 0: committed 1,204   partition 1: committed 988   partition 2: committed 1,377
group "analytics"   partition 0: committed 1,190   partition 1: committed 980   partition 2: committed 1,377
log end offsets     partition 0: 1,210             partition 1: 991             partition 2: 1,377

lag(billing) = (1,210 − 1,204) + (991 − 988) + 0 = 9 records

Consumer lag — log end offset minus committed offset, summed over partitions — is the single most important metric of a consuming application. Rising lag means the consumers cannot keep up or have stopped.

Rebalancing

When a member joins or leaves, or the topic's partition count changes, the group rebalances: the group coordinator (a broker chosen by hashing the group id) drives a protocol in which the members agree on a new assignment. During a classic eager rebalance every member gives up all its partitions and receives new ones, which pauses the whole group for seconds; the cooperative sticky assignor available since 2.4 only moves the partitions that must move. Kafka 4.0 introduced a new group protocol (KIP-848) that moves the assignment logic to the coordinator and makes rebalances incremental by design.

A member is considered dead when it stops sending heartbeats for session.timeout.ms or when it fails to call poll() within max.poll.interval.ms. The second is the one that bites: a consumer that takes six minutes to process one batch is kicked out of the group, its partitions are reassigned, its uncommitted work is redone elsewhere, and when it finally commits it fails with a rebalance error. Sizing batches and timeouts together is a recurring operational task.

Cluster metadata: the KRaft controller

Someone has to know which brokers exist, which partitions live where, who leads each partition and which replicas are in sync, and someone has to make decisions when a broker disappears. That someone is the controller.

Until Kafka 2.8, that state lived in ZooKeeper, a separate coordination service; one broker was elected controller through a ZooKeeper lock and pushed metadata changes to the others. It worked, but it meant running and securing two distributed systems, it limited the number of partitions a cluster could handle because controller failover had to reload everything from ZooKeeper, and it created subtle divergence between what ZooKeeper held and what brokers believed.

KRaft (Kafka Raft) replaces it with Kafka's own machinery. A small quorum of controller nodes — three or five — runs the Raft consensus protocol over an internal metadata log, __cluster_metadata. The elected leader of that quorum is the active controller; the others are hot standbys with the full metadata already in memory, so failover is near instant. Brokers fetch the metadata log like any consumer and apply changes in order. Controllers can run on dedicated nodes (process.roles=controller) or, for small clusters and development, on the same processes as brokers (process.roles=broker,controller).

ZooKeeper mode (legacy)KRaft mode
Metadata storeZooKeeper ensembleInternal Raft log on controller quorum
Controller failoverSeconds to minutes, reload from ZooKeeperMilliseconds, standby has state in memory
Partitions per clusterTens of thousandsMillions
Systems to operateTwoOne
StatusDeprecated in 3.5, removed in 4.0Production-ready since 3.3, the only mode in 4.0

For an application developer, KRaft changes nothing in the client API. For an operator, it removes an entire system from the deployment and makes bootstrap.servers plus a controller quorum the whole story.

Beyond the log: Connect and Streams

Two components ship with Kafka and reuse everything above. Kafka Connect is a framework of source and sink connectors that copy data between Kafka and external systems — databases through change data capture, object storage, search engines — with offsets, retries and scaling handled for you. Kafka Streams is a Java library for processing records topic-to-topic with joins, windows and state stores backed by compacted changelog topics, running inside your own application with no extra cluster. Both are why Kafka is called a platform rather than a broker, and both depend on the log semantics described here: replayable partitions, per-key ordering and compaction.

Designing a topic

Putting the model to work means answering a handful of questions before kafka-topics.sh --create.

  • Key. Choose the key by the ordering you need: an order id keeps one order's events in sequence; a customer id keeps everything about a customer in sequence but concentrates hot customers on one partition. A null key maximises spread and abandons ordering.
  • Partitions. Start from the target throughput divided by what one consumer can process, round up to a multiple of the broker count, and leave headroom — adding partitions later remaps keys. Very high counts cost memory, file handles and longer leader elections.
  • Replication factor and min.insync.replicas. 3 and 2 for anything that matters; 1 only for throwaway development data.
  • Retention. Long enough to rebuild a downstream consumer from scratch or to cover the longest outage you want to survive; compaction if the topic represents state rather than history.
  • Schema. Kafka does not care what the bytes are, so a schema registry with Avro, Protobuf or JSON Schema, and a versioning rule for compatible changes, is the difference between a durable contract and a pile of undocumented JSON.

Common mistakes

  • Expecting queue semantics. No per-record acknowledgement, no dead-letter queue, no redelivery; a failed record must be handled by the consumer (skip, retry, or write to an error topic). Teams that want those features on top of Kafka build them or use a queue.
  • One partition "to keep things ordered". It works until the day one consumer cannot keep up, and then the only fix is a repartition that changes ordering anyway. Order per key from the start.
  • Adding partitions to a keyed topic in production. Keys hash to different partitions afterwards; per-key ordering is broken across the change and any compacted state that assumed key locality is now inconsistent.
  • Replication factor 3 with min.insync.replicas 1. A single broker's write is acknowledged as durable. Set it to 2.
  • Ignoring lag. Rising consumer lag is the earliest signal of a stalled consumer, an under-provisioned group or a poison record. Alert on it.
  • Treating the metadata topic or controllers as optional. In KRaft mode, losing a majority of controllers stops leader elections and metadata changes cluster-wide. Run three or five controllers on separate failure domains.

Key takeaways

  • Kafka is a partitioned, replicated, retained log; consumers track offsets, and nothing is deleted because it was read.
  • Ordering and parallelism are both per partition, and the record key decides the partition, so key choice is the central design decision.
  • Segments and retention (delete or compact) govern storage; compaction keeps the latest value per key and makes a topic a rebuildable snapshot.
  • Leader replication with an in-sync replica set, acks=all and min.insync.replicas=2 on a replication factor of 3 is the standard durability configuration.
  • Consumer groups share partitions and commit offsets to __consumer_offsets; rebalances and poll timeouts are where consumer applications most often go wrong.
  • KRaft controllers hold cluster metadata in an internal Raft log; ZooKeeper is deprecated in 3.x and gone in 4.0.

Sources and further reading

Frequently asked questions

Is a Kafka topic ordered?
A partition is ordered; a topic with more than one partition is not. Records with the same key go to the same partition (with the default partitioner and a fixed partition count), so ordering holds per key. If you need global order you need a single partition, which caps throughput at what one broker and one consumer can do.
How many partitions should a topic have?
Enough to spread the load across brokers and to give the largest consumer group the parallelism it needs, since one partition can be read by at most one consumer in a group. Start with a small multiple of the broker count — six or twelve for a three-broker cluster — and remember that partitions can be added but not removed, and adding them changes which partition a key maps to.
Does Kafka delete a record once it has been consumed?
No. Records are removed by retention policy — after a configured age or when the partition exceeds a size — or replaced by a newer record with the same key under compaction. Consumption only moves a consumer group's offset. That is what makes it possible for several groups to read the same data and for any group to rewind.
Do I still need ZooKeeper?
Not for new clusters. KRaft mode, in which a quorum of Kafka controllers stores cluster metadata in an internal log, has been production-ready since Kafka 3.3 and is the only mode in Kafka 4.0. Existing ZooKeeper-based clusters on 3.x should be migrated with the documented migration procedure before upgrading to 4.x.