Delivery & Producers

Solution and explanation

Answer: no

Kafka **does not delete a record when it is consumed**. Records stay in the log until **retention** (time or size) or **compaction** removes them. Consuming only moves the consumer group's committed offset forward.

Why Kafka works this way

• **Multiple independent readers:** many consumer groups read the same records at different speeds. Deleting on read would make that impossible.
• **Replay:** after a bug fix you can reset offsets and reprocess days of events. A new service can be bootstrapped from history.
• **Performance:** no per-message delete bookkeeping; deleting whole segment files is cheap.

Topic orders (retention 7 days)
   [day1][day2][day3][day4][day5][day6][day7] ← new
     ↑ group analytics at day1     ↑ group billing at day6
Records older than 7 days are removed regardless of these positions.

Compare with traditional queues

Deleting data deliberately

• Lower retention.ms for the topic.
• Compacted topics: send a **tombstone** (key with null value) to delete a key.
• Admin API deleteRecords truncates a partition up to an offset.
• For personal data (GDPR/DPDP Act), design for deletion: use compaction with tombstones, or crypto-shredding (encrypt per user and delete the key).

Summary

No. Consumption only advances offsets; records are removed by retention or compaction, which enables multiple consumer groups and replay.

Solution and explanation

Model

• **Traditional queue (JMS, ActiveMQ, SQS, RabbitMQ classic queues):** the broker pushes or hands out messages, tracks each message's state, and removes it after a consumer acknowledges it. Each message goes to one consumer.
• **Kafka:** a durable, partitioned **log**. The broker stores records in order; consumers pull and track their own position (offset). Records remain until retention.

Comparison

Example

• **Sending 10,000 PDF resume-parsing jobs to workers:** a task queue (RabbitMQ, SQS, Laravel queue) fits: each job processed once, retried individually, no need to keep it.
• **Publishing every application.submitted event for notifications, analytics, ATS scoring and search:** Kafka fits: many consumers, history, ordering per candidate.

Kafka 4.0 share groups

KIP-932 ("Queues for Kafka") adds share groups with per-record acknowledgement and multiple consumers per partition, narrowing the gap for queue-style workloads.

Summary

A queue distributes individual messages to workers and deletes them; Kafka is a retained, replayable log read by many independent groups, built for high-throughput event streams.

Solution and explanation

Different designs

• **RabbitMQ** is a **smart broker, simple consumer** message broker (AMQP). Producers publish to **exchanges**; routing rules (direct, topic, fanout, headers) deliver copies to **queues**; consumers receive pushed messages and acknowledge each one.
• **Kafka** is a **simple broker, smart consumer** distributed log. Brokers append and store; consumers pull and manage offsets.

Comparison

When to choose which

Choose **Kafka** when:

• Many services need the same events (fan-out) and may need replay.
• Volume is high (clickstream, telemetry, CDC, logs).
• You do stream processing or event sourcing.

Choose **RabbitMQ** when:

• You need a task queue with per-message retries, priorities, delays and complex routing.
• Request/reply messaging or low-latency work distribution at moderate volume.
• The team wants simpler operations for a small system.

Banking example

Card transactions streamed to fraud detection, ledger projection and analytics: Kafka. Sending a "generate statement PDF" job to a pool of workers with retries: RabbitMQ (or a job queue).

Summary

RabbitMQ excels at flexible routing and per-message task queues; Kafka excels at high-throughput, retained, replayable event streams read by many consumers.

Solution and explanation

What it controls

acks decides **how many replicas must confirm a write before the leader tells the producer it succeeded**. It is the main producer setting for the trade-off between durability and latency.

Since Kafka 3.0, the default is **acks=all** with **idempotence enabled**.

props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120000);

Interaction with min.insync.replicas

acks=all alone waits for the *current* ISR, which could be only the leader. Setting topic min.insync.replicas=2 makes the broker reject writes when fewer than two replicas are in sync, guaranteeing at least two copies.

Latency

acks=all adds the time for followers to fetch the record (typically a few milliseconds in one data centre). Batching (linger.ms, batch.size) amortises this, so throughput with acks=all is still very high.

Choosing

• Payments, orders, ledger events: acks=all.
• Metrics or logs where small loss is acceptable and latency matters most: acks=1 (or even 0 for fire-and-forget telemetry).

Summary

acks sets how many replicas must have a record before the producer is told it is written: 0 (none), 1 (leader), all (every in-sync replica, the durable default).

Solution and explanation

Meaning

With acks=0 the producer **does not wait for any acknowledgement**. It writes the request to the network socket and immediately considers the record sent. The broker sends no response.

Consequences

• **Lowest latency and highest throughput** from the producer's point of view.
• **No delivery guarantee:** if the leader is down, the network drops the packet, the broker rejects the record (too large, wrong partition) or crashes before writing, the producer never knows.
• **Retries do nothing useful**, because no error is ever received.
• The returned RecordMetadata offset is -1 (unknown).
• **Idempotence cannot be enabled** (it requires acks=all).

props.put(ProducerConfig.ACKS_CONFIG, "0");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, false);

When it is acceptable

Only for data where losing some records does not matter and volume is huge:

• High-frequency IoT sensor readings where the next reading arrives in a second.
• Application metrics or debug logs sampled anyway.
• Real-time location pings where only the latest matters.

When it is wrong

Anything with business value: orders, payments, job applications, audit events, notifications. A silent loss there causes missing money or angry users with no trace.

Summary

acks=0 is fire-and-forget: fastest, but the producer never learns about failures, so records can be lost silently; use it only for disposable telemetry.

Solution and explanation

Meaning

With acks=1 the producer receives success as soon as the **partition leader has written the record to its local log**. The leader does not wait for followers to copy it.

The risk window

1. Producer → Leader (broker 1) writes offset 500 → replies "OK"
2. Broker 1 crashes before followers fetched offset 500
3. Broker 2 (follower, has up to 499) becomes leader
4. Offset 500 is gone, although the producer was told it succeeded

When broker 1 comes back it truncates its log to match the new leader, so the record is permanently lost. No exception is ever raised to the producer.

Characteristics

props.put(ProducerConfig.ACKS_CONFIG, "1");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, false);   // must be disabled explicitly in Kafka 3.x

When it was the default

Before Kafka 3.0, acks=1 was the default, which is why many older systems lost data silently during broker failures.

When it is reasonable

Logs, metrics, clickstream or analytics events where losing a few records during a rare broker crash is acceptable and lower latency matters.

Summary

acks=1 means only the leader confirmed the write: fast, but a leader crash before replication loses acknowledged records.

Solution and explanation

Meaning

With acks=all (equivalent to acks=-1) the leader replies success only after **every replica currently in the ISR** has the record. It is the most durable setting and the default since Kafka 3.0.

Why it is not enough on its own

"All in-sync replicas" can shrink to just the leader if followers fall behind. Then acks=all behaves like acks=1. The topic or broker setting **min.insync.replicas** fixes this by requiring a minimum ISR size:

# topic
replication.factor=3
min.insync.replicas=2
# producer
acks=all
enable.idempotence=true

Now a write succeeds only if at least two replicas have it. If the ISR drops to one, the producer gets NotEnoughReplicasException and retries until delivery.timeout.ms expires.

What you gain

• No loss of acknowledged records when one broker fails.
• Enables the **idempotent producer** and **transactions**.

What it costs

• Slightly higher latency: the leader waits for followers' next fetch (usually a few ms in one region; more across availability zones).
• Lower availability for writes when replicas are down.
• Throughput stays high with batching (linger.ms, batch.size, compression).

Example: payment events in Spring Boot

spring:
  kafka:
    producer:
      acks: all
      retries: 2147483647
      properties:
        enable.idempotence: true
        delivery.timeout.ms: 120000
        max.in.flight.requests.per.connection: 5

Summary

acks=all waits for all in-sync replicas; combined with min.insync.replicas=2 and replication factor 3 it guarantees acknowledged records survive a broker failure.

Solution and explanation

What it is

min.insync.replicas (topic or broker setting) is the **minimum number of in-sync replicas that must acknowledge a write** when the producer uses acks=all. If the ISR is smaller, the broker rejects the write with NotEnoughReplicasException (or NotEnoughReplicasAfterAppendException).

It does nothing for acks=0 or acks=1 producers.

Example with replication factor 3

So the cluster tolerates one failure without stopping writes and never accepts a write that exists on only one machine.

Common mistakes

• min.insync.replicas = replication.factor (e.g. 3 and 3): any single broker restart or rolling upgrade blocks all writes.
• min.insync.replicas=1 with acks=all: a write can be acknowledged by the leader alone, then lost.
• Forgetting internal topics: set transaction.state.log.min.isr=2 and offsets.topic.replication.factor=3 too.

Setting it

kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type topics --entity-name payments --add-config min.insync.replicas=2

Broker default: min.insync.replicas=2 in server.properties.

Monitoring

UnderMinIsrPartitionCount > 0 means some partitions are rejecting acks=all writes. Alert on it immediately.

Summary

min.insync.replicas sets the minimum number of replicas that must confirm an acks=all write; RF=3 with min ISR 2 is the standard choice, trading a little availability for guaranteed durability.

Solution and explanation

What it is

The producer does not send each record in its own network request. It groups records going to the **same partition** into a **batch** in memory (the RecordAccumulator), and a background **sender thread** ships batches to brokers. One request can carry batches for many partitions on that broker.

Why it matters

• Far fewer network round trips and disk writes.
• Compression works on whole batches, so the ratio is much better.
• Brokers write a batch as one unit.

Batching is the main reason Kafka reaches hundreds of thousands to millions of records per second per producer.

Settings

A batch is sent when it is full **or** linger.ms has passed, whichever comes first.

Tuning for throughput

batch.size=65536          # 64 KB
linger.ms=10
compression.type=lz4
buffer.memory=67108864

Latency rises by a few milliseconds while throughput and compression improve greatly.

Sticky partitioner

For records without a key, the producer sticks to one partition until the batch is sent, then switches. This produces fuller batches than strict round-robin.

Back-pressure

If brokers are slow and the buffer fills, send() blocks up to max.block.ms, then throws TimeoutException. This protects the application from running out of memory.

Summary

Producer batching groups records per partition and sends them together, controlled mainly by batch.size and linger.ms; bigger batches mean higher throughput and better compression at a small latency cost.

Solution and explanation

What it does

linger.ms is how long the producer **waits for more records to fill a batch** before sending it, even if the batch is not full. It trades a little latency for larger batches.

• linger.ms=0: send as soon as the sender thread is free (still batches records that arrived meanwhile).
• linger.ms=5 (default in Kafka 4.0): wait up to 5 ms.
• linger.ms=20–100: very high throughput pipelines.

A batch is sent at whichever comes first: batch.size reached or linger.ms expired.

Example effect

Producer sending 50,000 small records per second to one partition:

Fewer requests reduce CPU on producer and broker, improve compression and increase throughput.

Choosing

• Latency-sensitive (an OTP request event): linger.ms 0–5.
• Bulk events (clickstream, logs, CDC): 10–50 with batch.size 64–256 KB and compression.
• Low traffic: linger adds up to linger.ms of latency per record but gives little benefit; keep it small.

props.put(ProducerConfig.LINGER_MS_CONFIG, 10);
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 64 * 1024);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");

Related

flush() sends all buffered records immediately (useful at shutdown or before committing work), and close() flushes and releases resources.

Summary

linger.ms deliberately delays sending to build bigger batches; a few milliseconds usually give large throughput gains with negligible latency impact.