Kafka Fundamentals

Solution and explanation

What it is

Apache Kafka is a **distributed event-streaming platform**. Applications write events (records) to it, Kafka stores them durably on disk in an ordered, replicated log, and any number of applications read them, at their own pace, now or later. It was built at LinkedIn in 2011 and is now used by banks, Uber, Swiggy, Zerodha, Netflix and most large backends.

Kafka combines three capabilities:

• **Publish and subscribe** to streams of records (like a message system).
• **Store** streams durably and fault-tolerantly for as long as configured (days, or forever).
• **Process** streams as they arrive (Kafka Streams, ksqlDB, Flink).

Core vocabulary

Why it is fast

• Sequential disk writes and reads (append-only log) instead of random I/O.
• Relies on the OS page cache and **zero-copy** (sendfile) to send data from disk to network.
• Batching and compression of records.
• Horizontal scaling through partitions spread over brokers.

Example

producer.send(new ProducerRecord<>("orders.placed", orderId, orderJson));

The inventory, payment, notification and analytics services each read orders.placed independently.

Summary

Kafka is a durable, replicated, partitioned commit log that lets many systems publish and consume event streams at very high throughput with ordering per partition.

Solution and explanation

Main use cases

1. **Event-driven microservices:** services publish domain events (OrderPlaced, PaymentCaptured) instead of calling each other synchronously. Producers do not need to know who consumes.
2. **Messaging/decoupling:** absorb traffic spikes (a sale on Flipkart) so downstream services process at their own rate.
3. **Activity tracking and clickstreams:** page views, searches, job applications sent to analytics in real time (the original LinkedIn use case).
4. **Log and metrics aggregation:** collect logs from thousands of servers into one pipeline (to Elasticsearch, S3).
5. **Change data capture (CDC):** Debezium reads MySQL/Postgres binlogs and publishes every row change, keeping search indexes, caches and data warehouses in sync.
6. **Stream processing:** fraud detection on card transactions, surge pricing in ride hailing, real-time dashboards (Kafka Streams, Flink).
7. **Data integration pipelines:** Kafka Connect moves data between databases, S3, Elasticsearch, Snowflake.
8. **Event sourcing and audit logs:** the log is the system of record of what happened; state can be rebuilt by replaying.
9. **IoT and telemetry:** vehicle GPS updates every few seconds (Uber/Ola driver locations).

Example: a job portal

When Kafka is not the right tool

• Simple request/response where the caller needs an answer now (use REST/gRPC).
• Small apps with low volume (a database table plus a job queue is simpler).
• Complex per-message routing and priorities (RabbitMQ fits better).

Summary

Kafka is used for decoupling services, real-time analytics, log aggregation, CDC, stream processing and data integration: anywhere many systems need the same stream of events reliably.

Solution and explanation

What it is

A broker is a single **Kafka server process**. It receives records from producers, writes them to partition log files on disk, serves reads to consumers, and replicates partitions to and from other brokers. Each broker has a unique node.id (broker.id in older versions).

Responsibilities

• Hosting partitions: each partition replica is a directory of segment files (00000000000000000000.log, .index, .timeindex).
• Acting as **leader** for some partitions (handles all reads and writes for them) and **follower** for others (copies data from leaders).
• Handling client requests: produce, fetch, metadata, offset commits (the __consumer_offsets topic lives on brokers too).
• Enforcing retention: deleting or compacting old segments.

Clients find brokers through bootstrap servers

bootstrap.servers=broker1:9092,broker2:9092,broker3:9092

The client connects to any one, fetches cluster **metadata** (which broker leads which partition), then talks directly to the right leaders. Listing a few brokers is enough; it does not need all of them.

Sizing

A broker can handle thousands of partitions and hundreds of MB/s, limited by disk throughput, network and the page cache. Production clusters usually start with **at least three brokers** so replication factor 3 is possible and one broker can fail.

Key broker settings

log.dirs=/data/kafka
num.partitions=6
default.replication.factor=3
min.insync.replicas=2
log.retention.hours=168

Summary

A broker is one Kafka server that stores partition replicas, leads some partitions, follows others, and serves producers and consumers; a cluster is several brokers working together.

Solution and explanation

What it is

A Kafka cluster is a group of brokers that together store and serve topics. Partitions of each topic are spread across brokers, and each partition is replicated to several brokers, so the cluster provides **scalability** (more brokers, more throughput and storage) and **fault tolerance** (a broker can fail without data loss).

Components

• **Brokers:** store data and serve clients.
• **Controller:** one elected component that manages cluster metadata: which broker leads which partition, broker membership, topic creation and leader elections.
• Older Kafka: metadata in **ZooKeeper**, one broker acts as controller.
• Kafka 3.3+ / 4.0: **KRaft** mode, a quorum of controller nodes stores metadata in an internal Raft log; ZooKeeper is removed in 4.0.

Example layout

Topic payments with 3 partitions and replication factor 3 on 3 brokers:

Leaders are spread so every broker shares the load. If Broker 2 dies, the controller makes Broker 3 or 1 the leader of P1.

Scaling a cluster

Adding brokers does not move existing partitions automatically; use kafka-reassign-partitions.sh (or Cruise Control) to rebalance replicas onto new brokers.

Multi-cluster setups

• One cluster per region or environment (dev, staging, prod).
• **MirrorMaker 2** or Confluent Cluster Linking copy topics between clusters for disaster recovery or geo-distribution.

Summary

A cluster is a set of brokers plus a controller (ZooKeeper-based or KRaft) that spreads and replicates partitions for throughput and fault tolerance.

Solution and explanation

What it is

A topic is a **named, logical stream of records** of one kind, like a table name for events: orders.placed, payments.completed, user.signup. Producers write to a topic, consumers subscribe to it. Physically a topic is split into **partitions** stored on brokers.

Properties

• **Append-only:** records are added at the end and never modified.
• **Multi-subscriber:** any number of consumer groups can read the same topic independently.
• **Durable:** records stay until retention removes them (time, size) or compaction keeps only the latest per key, regardless of whether they were consumed.

Creating a topic

kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic payments.completed --partitions 6 --replication-factor 3 \
  --config retention.ms=604800000 --config min.insync.replicas=2

// Spring Kafka
@Bean
NewTopic paymentsTopic() {
    return TopicBuilder.name("payments.completed").partitions(6).replicas(3).build();
}

Naming and design

• Use a convention: <domain>.<entity>.<event> such as banking.transfer.completed.
• One topic per event type (or per entity with a type header); avoid one giant topic for everything.
• Set **partitions** for parallelism, **replication factor** for durability, **retention** for how long data is needed.
• Disable auto.create.topics.enable in production so typos do not create topics.
• Define a schema (Avro, Protobuf, JSON Schema with a Schema Registry) so producers and consumers agree on the format.

Summary

A topic is a named, durable, append-only stream of related events, split into partitions and readable by many independent consumer groups.

Solution and explanation

What it is

A partition is an **ordered, immutable, append-only log** that holds part of a topic's records. A topic with 6 partitions has six independent logs. Each record in a partition gets a sequential **offset** (0, 1, 2, …).

Topic orders, partition 0:  [0][1][2][3][4][5] ← new records appended here
Topic orders, partition 1:  [0][1][2][3]
Topic orders, partition 2:  [0][1][2][3][4]

How records choose a partition

• **With a key:** partition = murmur2(key) % numPartitions. Same key, same partition, so events for one customer or account stay in order.
• **Without a key:** the default partitioner uses a **sticky** strategy: it fills a batch for one partition, then moves to another, spreading load while keeping batches large.
• **Explicit partition** or a custom Partitioner can also be used.

On disk

Each partition replica is a directory of **segments** (files of e.g. 1 GB). Only the newest segment is written; older segments are deleted or compacted according to retention. Index files map offsets and timestamps to file positions for fast lookups.

Guarantees

• **Ordering** is guaranteed only within a partition, not across partitions.
• One partition is read by **at most one consumer in a group** at a time.

Partitions and replicas

Each partition has a leader replica and follower replicas on other brokers. Clients read and write the leader (followers can serve reads with rack-aware fetching).

Summary

A partition is the unit of ordering, storage and parallelism in Kafka: an append-only log of records with sequential offsets, chosen by hashing the record key.

Solution and explanation

Partitions are how Kafka scales

1. **Parallel consumption:** within a consumer group, each partition is assigned to one consumer. 12 partitions allow up to 12 consumers to work in parallel; a 13th would sit idle.
2. **Parallel writes and storage:** partitions are spread across brokers, so writes, disk space and network load are shared. A topic is not limited by one machine.
3. **Ordering by key:** records with the same key always go to the same partition, giving per-key order (all events of account ACC-42 in sequence) while different keys proceed in parallel.
4. **Fault tolerance unit:** each partition is replicated independently and has its own leader, so leadership is spread over the cluster.
5. **Retention and compaction** happen per partition segment.

Example

A payments topic receives 50,000 events per second. One consumer handles about 5,000 per second. You need at least 10 consumers, so at least 10 partitions; you might choose 24 to allow for growth.

Trade-offs of too many partitions

• More open files and memory on brokers, more metadata.
• Longer leader elections and recovery after a broker failure (less of an issue with KRaft).
• Producers spread data thinner: smaller batches, more requests, higher latency.
• **Increasing** partitions later changes hash(key) % n, so keys move to different partitions and ordering for a key can break during the change. Partitions can never be decreased.

Rule of thumb

partitions ≥ max(target throughput / per-consumer throughput, target throughput / per-producer throughput), plus headroom. Many teams start with 6–12 for normal topics and measure.

Summary

Partitions give Kafka horizontal scalability, parallelism and per-key ordering; choose the count from throughput needs and plan ahead because increasing it remaps keys.

Solution and explanation

What it is

An offset is a **64-bit sequential number** that identifies the position of a record within a partition. The first record in a partition has offset 0, the next 1, and so on. Offsets never change and are never reused, even after old records are deleted by retention.

Partition 3:  offset 0 | 1 | 2 | 3 | 4 | 5 | 6 ...
                                  ↑ committed offset of group "billing" = 4
                                          ↑ log end offset (next write) = 7

Two kinds of offsets that matter

• **Record offset:** where a record sits in the log.
• **Consumer (committed) offset:** for each consumer group and partition, the offset of the **next record to read**. Stored in the internal topic __consumer_offsets. If a consumer restarts, it resumes from the committed offset.

Other positions: **log-start offset** (oldest record still kept) and **log-end offset / high watermark** (the latest offset replicated to all in-sync replicas; consumers can read only up to the high watermark).

Using offsets

consumer.seek(new TopicPartition("payments", 3), 1500);   // replay from offset 1500
consumer.seekToBeginning(partitions);                     // reprocess everything kept

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group billing \
  --reset-offsets --to-datetime 2026-10-01T00:00:00.000 --topic payments --execute

Resetting offsets lets you **replay** events after fixing a bug, which a traditional queue cannot do.

auto.offset.reset

When a group has no committed offset (new group) or the offset was deleted: earliest reads from the beginning, latest (default) reads only new records, none throws an error.

Summary

An offset is a record's sequential position in a partition; consumer groups commit offsets to remember progress, which enables restarts and replay.

Solution and explanation

Answer: no

An offset is unique **only within a partition**. Every partition starts counting from 0 independently, so offset 100 exists in partition 0, partition 1, partition 2, and so on.

orders-0: 0 1 2 3 ... 100 ...
orders-1: 0 1 2 3 ... 100 ...
orders-2: 0 1 2 3 ... 100 ...

Uniquely identifying a record

A record is identified by the triple **(topic, partition, offset)**:

for (ConsumerRecord<String, String> r : records) {
    String id = r.topic() + "-" + r.partition() + "-" + r.offset();   // e.g. orders-2-100
}

This triple is a good idempotency key for consumers that must not process the same record twice, as long as the topic is never recreated. For business-level deduplication prefer a **business id** in the payload (order id, transaction reference), because the same business event could be produced twice and land at two different offsets.

Consequences

• Consumer progress is tracked **per partition**: a group commits one offset for each partition it reads.
• There is no single "position in the topic"; comparing offsets across partitions is meaningless. Ordering across partitions is not defined either.
• Lag is calculated per partition (log-end offset − committed offset) and summed for the topic.
• If a topic is deleted and recreated, offsets start again at 0.

Summary

Offsets are sequential per partition, not per topic; use (topic, partition, offset) to identify a record, and a business id for deduplication.

Solution and explanation

Producers

A producer **writes records** to topics. It serialises key and value, chooses the partition (by key hash or the sticky partitioner), batches records per partition, optionally compresses them and sends them to the partition leader. It can wait for acknowledgements (acks) and retries on transient errors.

Properties p = new Properties();
p.put("bootstrap.servers", "localhost:9092");
p.put("key.serializer", StringSerializer.class.getName());
p.put("value.serializer", StringSerializer.class.getName());
p.put("acks", "all");
p.put("enable.idempotence", "true");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(p)) {
    producer.send(new ProducerRecord<>("payments", "ACC-42", "{\"amount\":500}"),
        (meta, ex) -> {
            if (ex != null) log.error("send failed", ex);
            else log.info("written to {}-{}@{}", meta.topic(), meta.partition(), meta.offset());
        });
}

send is asynchronous; the callback reports success or failure.

Consumers

A consumer **reads records** by polling partitions it is assigned, deserialises them, processes them and commits offsets to record progress. Consumers **pull** data (Kafka does not push), so each reads at its own speed.

p.put("group.id", "billing-service");
p.put("enable.auto.commit", "false");
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(p)) {
    consumer.subscribe(List.of("payments"));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> r : records) process(r);
        consumer.commitSync();
    }
}

Spring Kafka

kafkaTemplate.send("payments", accountId, event);

@KafkaListener(topics = "payments", groupId = "billing-service")
public void onPayment(PaymentEvent event) { billing.record(event); }

Decoupling

Producers do not know who consumes; consumers do not know who produced. New consumers can be added later and even replay history.

Summary

Producers serialise, partition, batch and send records with acknowledgements; consumers poll assigned partitions, process records and commit offsets, each at their own pace.