KRaft & Scaling

Solution and explanation

What happens

A follower that has not caught up with the leader's log end for longer than replica.lag.time.max.ms (default 30 s) is **removed from the ISR**. It keeps fetching and replicating; once it catches up, the leader adds it back.

Causes

• Broker overloaded (CPU, disk I/O saturation, too many partitions).
• Slow or failing disk, network congestion between brokers or availability zones.
• Long garbage collection pauses.
• Broker restarting and catching up after downtime.
• Under-sized num.replica.fetchers for a high partition count.

Effects

• The partition is **under-replicated**: fewer copies of new data.
• With acks=all, the producer now needs only the remaining ISR members, so writes continue as long as ISR ≥ min.insync.replicas.
• If ISR drops below min.insync.replicas, acks=all writes fail with NotEnoughReplicasException.
• The lagging replica **cannot become leader** (with clean leader election), so if the leader also fails and no other ISR member exists, the partition goes offline.

Detection

• Metric UnderReplicatedPartitions > 0 and IsrShrinksPerSec / IsrExpandsPerSec.
• kafka-topics.sh --describe --under-replicated-partitions.
• Replica fetcher lag metrics (ReplicaFetcherManager MaxLag).

Fixes

• Find the slow broker (disk latency, CPU, network) and fix or replace it.
• Rebalance partition leadership and replicas across brokers (Cruise Control).
• Increase num.replica.fetchers, add brokers, reduce partitions per broker.
• Throttle reassignments so recovery does not overload the cluster.

Summary

A lagging follower is removed from the ISR (reducing redundancy and possibly blocking acks=all writes), keeps replicating, and rejoins once caught up; monitor under-replicated partitions and fix the slow broker.

Solution and explanation

What it is

Unclean leader election allows an **out-of-sync replica** (not in the ISR) to become leader when **no in-sync replica is available**. It is controlled by unclean.leader.election.enable (default **false** since Kafka 0.11).

The trade-off

Leader B1 has offsets 0..1000 (ISR = {B1})
Follower B2 has 0..950, out of ISR
B1 dies permanently
unclean=true  → B2 leads; offsets 951..1000 are lost forever; new writes reuse those offsets
unclean=false → partition unavailable until B1 is back

Consistency issues beyond loss

Consumers that already read offsets 951–1000 from the old leader saw data that no longer exists; their committed offsets may point past the new log end, causing resets or skipped data.

When enabling might be considered

Only for data where availability matters more than completeness (metrics, logs) and the business accepts loss. Never for payments, orders or ledgers.

Manual last resort

An operator can trigger an unclean election for a specific partition during an emergency:

kafka-leader-election.sh --bootstrap-server localhost:9092 --election-type unclean --topic metrics --partition 3

Summary

Unclean leader election promotes an out-of-sync replica when no ISR member is alive, restoring availability at the cost of losing acknowledged data; keep it disabled for important topics.

Solution and explanation

What it is

The controller is the component that **manages cluster metadata and coordinates cluster-wide changes**. Only one controller is active at a time.

Responsibilities

• Tracking which brokers are alive (registrations, heartbeats).
• **Electing partition leaders** when a broker fails or shuts down, and updating ISR changes.
• Creating and deleting topics, adding partitions, partition reassignments.
• Propagating metadata to all brokers so clients get correct routing.
• Managing configuration changes, ACLs, quotas and feature versions.

Two generations

ZooKeeper mode (legacy, removed in Kafka 4.0)

• Metadata lived in ZooKeeper.
• One broker was elected controller (via an ephemeral ZooKeeper node).
• On controller failover, the new controller had to **load all metadata from ZooKeeper**, which took a long time with hundreds of thousands of partitions.

KRaft mode (default since 3.3 for new clusters, only mode in 4.0)

• A small quorum of **controller nodes** (usually 3 or 5) store metadata in an internal Raft-replicated log (__cluster_metadata).
• One is the active controller (Raft leader); the others are hot standbys with the full metadata in memory, so **failover is near-instant**.
• Brokers fetch metadata changes from the log like consumers.

process.roles=controller
node.id=1
controller.quorum.voters=1@ctrl1:9093,2@ctrl2:9093,3@ctrl3:9093

Why it matters operationally

Controller health determines how fast the cluster recovers from broker failures. Metrics: ActiveControllerCount must be exactly 1 across the cluster; watch leader election rates and metadata lag.

Summary

The controller manages metadata and leader elections; in KRaft mode a Raft quorum of controller nodes replaces ZooKeeper, giving faster failover and simpler operations.

Solution and explanation

What it is

**KRaft (Kafka Raft)** is Kafka's built-in metadata management system, introduced by KIP-500. It **removes the dependency on Apache ZooKeeper**: cluster metadata is stored in an internal topic (__cluster_metadata) replicated with a Kafka-specific implementation of the **Raft consensus protocol** among controller nodes.

Timeline

• Kafka 2.8 (2021): early access.
• Kafka 3.3 (2022): production-ready for new clusters.
• Kafka 3.5–3.9: migration path from ZooKeeper, ZooKeeper deprecated.
• **Kafka 4.0 (2025): ZooKeeper removed; KRaft is the only mode.**

Why it was built

Roles

process.roles=broker              # data node
process.roles=controller          # metadata quorum member
process.roles=broker,controller   # combined mode (small or dev clusters)
controller.quorum.bootstrap.servers=ctrl1:9093,ctrl2:9093,ctrl3:9093

Production clusters usually run **3 dedicated controllers** (tolerates one failure) or 5 (tolerates two).

Setting up a new cluster

KAFKA_CLUSTER_ID=$(kafka-storage.sh random-uuid)
kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/server.properties
kafka-server-start.sh config/server.properties

Summary

KRaft is Kafka's Raft-based internal metadata quorum that replaced ZooKeeper, giving one system to operate, faster controller failover and much larger cluster scale.

Solution and explanation

Answer: no

KRaft is **not** a separate product or a competitor to Kafka. It is **part of Kafka itself**: the mode in which Kafka manages its own metadata using the Raft protocol instead of relying on ZooKeeper.

The confusion comes from the phrase "Kafka without ZooKeeper". What changed is only **where cluster metadata lives and how the controller is elected**:

Applications do not change: producers and consumers connect to brokers exactly as before (bootstrap.servers). Only cluster operators see the difference (no ZooKeeper to run; controller nodes configured with process.roles=controller).

What KRaft is an alternative to

**ZooKeeper** as Kafka's metadata system. Since Kafka 4.0, it is the only option; older clusters must migrate from ZooKeeper to KRaft (supported in-place through Kafka 3.9) before upgrading to 4.x.

Summary

KRaft is Kafka's own Raft-based metadata mode that replaced ZooKeeper; it is Kafka, not an alternative to it, and client applications work unchanged.

Solution and explanation

• Controllers reaching Raft consensus on cluster metadata.

Solution and explanation

• Spreading replicas across failure domains such as racks or zones.

Solution and explanation

• Use measured partitioning, brokers, replication, batching, compression and consumer capacity.

Solution and explanation

• Consider throughput, parallelism, capacity, growth, ordering and key distribution, then load test.

Solution and explanation

• A partition receiving disproportionate traffic because keys are skewed.