Messaging & Event Streaming
Installing and Operating Apache Kafka in KRaft Mode: Setup, CLI Tools and Monitoring
Intermediate7 min readUpdated
A Kafka cluster without ZooKeeper is a handful of Java processes and one properties file each, but the settings in that file decide whether the cluster survives a lost disk or a full one. This guide installs Kafka 3.x in KRaft mode as a single node and as a three-broker cluster, covers the command-line tools you use every day for topics, consumer groups and offsets, and explains the broker settings, metrics, security options and upgrade steps that matter in production.
What you will learn
- Format storage and start a KRaft broker or combined broker/controller from the tarball or a container
- Write server.properties for a three-broker, three-controller production cluster
- Create, inspect and reconfigure topics and reset consumer group offsets with the CLI tools
- Set the broker defaults that protect data: replication, min.insync.replicas, retention and topic creation
- Monitor the metrics that predict outages: under-replicated partitions, controller state, request latency and consumer lag
- Enable TLS, SASL and ACLs and plan a rolling upgrade or ZooKeeper migration
Kafka has a reputation for being hard to operate that dates from the ZooKeeper era, when every cluster meant two distributed systems, two sets of security configuration and a controller failover that could take minutes. In KRaft mode, which has been production-ready since Kafka 3.3 and is the only mode in 4.0, a cluster is a set of identical Java processes whose roles are set in one properties file. This guide installs Kafka 3.x that way, then covers the daily tooling and the production settings that the quickstart does not mention. All commands are for the Linux tarball; the container variant is shown where it differs.
Prerequisites and layout
A broker needs a JDK — 17 is the safe choice for 3.x and required for 4.x — a dedicated user, and a filesystem with room for log.dirs. Disk matters more than CPU: Kafka writes sequentially and reads through the page cache, so several ordinary disks or one fast volume per broker with the data directory on its own mount point is the usual setup. Give the JVM 4 to 6 GB of heap and leave the rest of the machine's memory to the page cache; a bigger heap makes Kafka slower, not faster.
sudo useradd --system --home /opt/kafka --shell /usr/sbin/nologin kafka
curl -LO https://archive.apache.org/dist/kafka/3.9.0/kafka_2.13-3.9.0.tgz
sudo tar -xzf kafka_2.13-3.9.0.tgz -C /opt
sudo ln -sfn /opt/kafka_2.13-3.9.0 /opt/kafka
sudo mkdir -p /var/lib/kafka && sudo chown -R kafka:kafka /var/lib/kafka /opt/kafka_2.13-3.9.0
The 2.13 in the file name is the Scala version the broker was built with; it has no effect on Java clients. Inside the archive, bin/ holds the CLI scripts, config/kraft/ holds the KRaft sample configurations (server.properties for a combined node, broker.properties and controller.properties for separate roles), and libs/ the jars. Kafka 4.0 drops the kraft/ subdirectory because there is no other mode.
A single-node cluster
A KRaft node must be told which cluster it belongs to before it starts, which is what the storage format step does. It writes a meta.properties file into each log directory with the cluster id and node id; a broker refuses to start on a directory formatted for another cluster, which prevents the classic accident of a stray broker joining production.
# config/kraft/server.properties — combined broker and controller for development
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
log.dirs=/var/lib/kafka
num.partitions=3
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
KAFKA_CLUSTER_ID=$(bin/kafka-storage.sh random-uuid) # once per cluster; keep it
bin/kafka-storage.sh format -t "$KAFKA_CLUSTER_ID" -c config/kraft/server.properties
bin/kafka-server-start.sh config/kraft/server.properties # foreground; add -daemon to background
The same thing as a container, with the official image the project has published since 3.7:
docker run -d --name kafka -p 9092:9092 \
-e KAFKA_NODE_ID=1 \
-e KAFKA_PROCESS_ROLES=broker,controller \
-e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \
-e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
-e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
-e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \
-e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
-e KAFKA_LOG_DIRS=/var/lib/kafka/data \
-v kafka-data:/var/lib/kafka/data \
apache/kafka:3.9.0
Every property maps to an environment variable by upper-casing it, replacing dots with underscores and prefixing KAFKA_. The image formats storage itself on first start with a generated cluster id.
Verify the node with a topic and a round trip:
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic smoke --partitions 1 --replication-factor 1
echo "hello" | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic smoke
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic smoke --from-beginning --max-messages 1
A production cluster
A real cluster separates roles. Three controllers on small machines form the metadata quorum; three or more brokers on machines with real disks hold the data. Each node has its own node.id, unique across both roles.
# controller.properties on controller nodes 1, 2, 3 (change node.id on each)
process.roles=controller
node.id=1
controller.quorum.voters[email protected]:9093,[email protected]:9093,[email protected]:9093
listeners=CONTROLLER://:9093
controller.listener.names=CONTROLLER
log.dirs=/var/lib/kafka/metadata
# broker.properties on broker nodes 11, 12, 13 (change node.id and advertised host on each)
process.roles=broker
node.id=11
controller.quorum.voters[email protected]:9093,[email protected]:9093,[email protected]:9093
listeners=INTERNAL://:9092,EXTERNAL://:9094
advertised.listeners=INTERNAL://kafka-11.internal:9092,EXTERNAL://kafka-11.example.com:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:SASL_SSL,CONTROLLER:PLAINTEXT
inter.broker.listener.name=INTERNAL
controller.listener.names=CONTROLLER
log.dirs=/data/kafka-1,/data/kafka-2 # one directory per disk; partitions are spread across them
# Defaults that protect data
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
unclean.leader.election.enable=false
auto.create.topics.enable=false # topics are created deliberately, with the right settings
num.partitions=6
log.retention.hours=168
log.segment.bytes=1073741824
message.max.bytes=1048588
Two of these deserve emphasis. auto.create.topics.enable=false stops a misspelt topic name in a producer from silently creating a one-partition, under-replicated topic. And min.insync.replicas=2 is the broker-side half of the durability contract: with it, a producer using acks=all is refused when fewer than two replicas are alive, rather than being told that a record is safe when it exists on one machine.
Kafka 3.9 added dynamic quorum membership (controller.quorum.bootstrap.servers instead of a static controller.quorum.voters list, plus kafka-metadata-quorum.sh add-controller), which lets you replace a controller without restarting the others. The static list still works and is simpler to reason about for a first cluster.
Run each node under systemd with Restart=on-failure, a generous LimitNOFILE (Kafka holds a file handle per segment and index), and KAFKA_HEAP_OPTS="-Xms4g -Xmx4g" in the environment.
Daily operations with the CLI
Every script under bin/ talks to the cluster through --bootstrap-server; there is no separate admin port. The four you use most:
Topics. Create with explicit settings, describe to see leaders and in-sync replicas, alter to add partitions or change configuration.
bin/kafka-topics.sh --bootstrap-server kafka-11.internal:9092 --create --topic orders \
--partitions 12 --replication-factor 3 --config min.insync.replicas=2 --config retention.ms=1209600000
bin/kafka-topics.sh --bootstrap-server kafka-11.internal:9092 --describe --topic orders
# Topic: orders Partition: 0 Leader: 11 Replicas: 11,12,13 Isr: 11,12,13
# A partition whose Isr is shorter than Replicas is under-replicated
bin/kafka-configs.sh --bootstrap-server kafka-11.internal:9092 --alter --entity-type topics \
--entity-name orders --add-config cleanup.policy=compact,delete
Consumer groups. Describe to see each partition's committed offset, log end offset and lag, and which member owns it; reset offsets to replay or skip.
bin/kafka-consumer-groups.sh --bootstrap-server kafka-11.internal:9092 --describe --group billing
# GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
# billing orders 0 1204 1210 6 billing-1-… /10.0.1.5 billing
# Rewind the whole group to two hours ago — the group must be stopped first
bin/kafka-consumer-groups.sh --bootstrap-server kafka-11.internal:9092 --group billing \
--topic orders --reset-offsets --to-datetime 2026-09-07T07:00:00.000 --execute
# Other targets: --to-earliest, --to-latest, --to-offset N, --shift-by -100, --by-duration PT1H
Console producer and consumer. For checking that data flows and for reading a topic by eye; --property print.key=true --property print.timestamp=true on the consumer and --property parse.key=true --property key.separator=: on the producer make keys visible.
Metadata quorum. bin/kafka-metadata-quorum.sh --bootstrap-server kafka-11.internal:9092 describe --status shows the active controller, the voters, their lag behind the leader and the current metadata version; it is the first command to run when the cluster misbehaves.
Two more commands earn their place in a runbook: kafka-reassign-partitions.sh moves replicas between brokers when you add a node or retire one (it does not rebalance automatically), and kafka-leader-election.sh restores preferred leaders after a broker returns so that traffic is spread evenly again.
Monitoring
Brokers expose hundreds of JMX metrics; a Prometheus JMX exporter or the vendor agent of your monitoring system turns them into dashboards. A short list predicts most incidents:
| Metric (JMX) | Healthy value | Meaning when it is not |
|---|---|---|
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions | 0 | A follower is behind or down; durability with acks=all is reduced and, at min.insync.replicas, producers will be refused |
kafka.controller:type=KafkaController,name=ActiveControllerCount | 1 across the cluster | 0 means no controller — no leader elections, no topic changes; more than 1 is a split |
kafka.server:type=ReplicaManager,name=OfflinePartitionsCount | 0 | Partitions with no leader; producers and consumers of them fail |
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce (p99) | Low tens of ms | Disk or replication latency; producers approach delivery.timeout.ms |
kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent | Above 0.3 | Broker threads saturated; add brokers or reduce partition count per broker |
kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs | Stable | Rising flush time means the disk is the bottleneck |
Consumer lag per group (kafka-consumer-groups.sh or records-lag-max on the consumer) | Stable or shrinking | Consumers are too slow, stopped, or stuck on a poison record |
Alert on disk usage of every log.dirs path as well: a broker whose data disk fills up takes the affected directory offline, and with a single directory that means the whole broker. Retention limits are your defence; size them against the disk with headroom for the largest topic's burst.
Security
A default Kafka listener is plaintext and unauthenticated. Securing it takes three layers, all set per listener:
- Encryption with TLS:
ssl.keystore.locationandssl.truststore.locationon the broker,SSLorSASL_SSLinlistener.security.protocol.map. - Authentication with SASL.
SCRAM-SHA-512stores credentials in the cluster's metadata (kafka-configs.sh --alter --add-config 'SCRAM-SHA-512=[password=...]' --entity-type users --entity-name orders),OAUTHBEARERintegrates with an identity provider, andGSSAPIwith Kerberos. mTLS with client certificates is a fourth option that needs no SASL at all. - Authorization with ACLs:
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizeron every node,super.usersfor operators, andkafka-acls.sh --add --allow-principal User:orders --operation Write --topic ordersfor applications. GrantReadon topics plusReadon the consumer group for consumers; forgetting the group ACL is the most common cause of aGroupAuthorizationException.
Keep the controller listener on a private network; it carries the metadata log and, in a separated deployment, does not need to be reachable by clients at all.
Upgrades and the ZooKeeper migration
Brokers are upgraded one at a time: stop the broker, install the new version, start it, wait until UnderReplicatedPartitions returns to zero, move on. Clients keep working throughout because the wire protocol is negotiated per connection. When every node runs the new version, raise the metadata version — bin/kafka-features.sh --bootstrap-server ... upgrade --metadata 3.9 — to unlock features the new release adds; until then the cluster behaves as the old version, and the step can be delayed as long as you like.
Clusters still on ZooKeeper must migrate before 4.0, which cannot read ZooKeeper at all. The documented procedure provisions a KRaft controller quorum in migration mode, lets it copy the metadata from ZooKeeper, restarts brokers one at a time in KRaft mode, and finally takes the controllers out of migration mode; there is a rollback path until that last step. Plan it as a project with a rehearsal on a copy of production, not as a maintenance window.
Common mistakes
advertised.listenersleft at localhost. The single most common "it works on the box but nowhere else" problem; set it to what clients can reach.- Replication factor 1 for the internal topics. The quickstart's
offsets.topic.replication.factor=1in production means one lost broker loses every consumer group's position. Set the three internal replication factors to 3. auto.create.topics.enableleft on. Typos become permanent under-replicated topics; someone eventually consumes from the wrong one.- Heap sized like a database. 4 to 6 GB is plenty; the page cache does the real work. Very large heaps mean long garbage-collection pauses that look like a dead broker to the controller.
- Adding a broker and expecting it to take load. Existing partitions stay where they are until you run
kafka-reassign-partitions.sh. - No lag alerting. A consumer that has been dead for a week is discovered when its retention window runs out and the data is gone.
Key takeaways
- KRaft removes ZooKeeper: format each node's storage with the cluster id, set
process.roles,node.idand the controller quorum, and start the process. - Development runs one combined node; production runs three dedicated controllers and three or more brokers, each with
advertised.listenersthat clients can actually reach. default.replication.factor=3,min.insync.replicas=2, replicated internal topics,auto.create.topics.enable=falseand explicit retention are the broker settings that keep data safe.kafka-topics.sh,kafka-consumer-groups.sh,kafka-configs.shandkafka-metadata-quorum.shcover most daily work, including resetting a group's offsets for replay.- Watch under-replicated and offline partitions, the active controller count, produce latency, handler idle percentage, disk usage and consumer lag.
- Secure listeners with TLS, SASL and ACLs; upgrade brokers one at a time and raise the metadata version afterwards; migrate any remaining ZooKeeper clusters before moving to 4.x.
Sources and further reading
Frequently asked questions
- How many controllers does a KRaft cluster need?
- Three for most clusters, five for very large or very critical ones. Raft needs a majority to make progress, so three controllers tolerate one failure and five tolerate two. A single controller is fine for development. Combined broker/controller nodes are acceptable for small clusters, but production deployments usually run controllers as separate, lightly loaded processes so that broker load cannot delay metadata decisions.
- Why can my clients connect from the host but not from other machines or containers?
- Because advertised.listeners tells clients where to connect for subsequent requests, and it defaults to the listeners value, often localhost. A client bootstraps successfully, receives localhost:9092 as the leader address, and then fails. Set advertised.listeners to a hostname or IP that every client can resolve and reach, with separate listeners for internal and external traffic if they differ.
- Can I shrink the number of partitions of a topic?
- No. Partitions can only be added. To reduce them, create a new topic with the desired count, copy the data with a consumer-producer pair or MirrorMaker, switch consumers and producers over, and delete the old topic. Because keys map to partitions by hash, the copy changes per-key ordering relative to the old topic.
- Do I need to upgrade clients when I upgrade brokers?
- No. The Kafka protocol is versioned and brokers accept requests from older clients; clients newer than the broker also work but will not use features the broker lacks. Upgrade brokers first, one at a time, then bump the metadata version with kafka-features.sh, then upgrade clients at your own pace.