Kafka Part 1 — Brokers, Topics, Partitions, Producers

Why this session matters

This is Session 10 of 48 in the Data Engineering track. It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

• Kafka mental model — distributed append-only log per topic-partition
• Brokers, controller, ZooKeeper-vs-KRaft — the metadata story
• Topics, partitions, offsets — the unit of parallelism
• Producer semantics — acks, idempotence, batching, compression
• Where consumers and replication fit (Part 2)

Pre-read (skim before the session)

• Confluent — Apache Kafka 101 (free course)
• Kafka design doc (official)
• Designing Data-Intensive Applications, Ch. 11
• Confluent KRaft mode docs

Deep dive

1. The one-line mental model

Kafka is a distributed, durable, append-only log partitioned across brokers. Producers append; consumers read at their own pace and track their own offset. That's it. Everything else is engineering around those primitives.

2. Why it won

• Decouples producers from consumers (multiple consumers can read the same topic).
• Durable by default — messages survive consumer restarts.
• Massive throughput — millions of events/sec on a moderate cluster.
• Becomes the event backbone for microservices, ETL, ML, search indexing.

Used at LinkedIn (origin), Uber, Netflix, Microsoft, Goldman Sachs — essentially everywhere with serious data movement.

3. Cluster topology

   producers          brokers (1…N)         consumers
   ---------         ---------------         ---------
      ▼                  ▼                       ▲
     (write)         topic A: partitions 0,1,2,3,4   (read)
                     topic B: partitions 0,1
                          │
                          ▼
                  controller (one elected broker;
                  KRaft uses Raft, replacing ZooKeeper)

Each partition is owned by one leader broker; reads and writes go through the leader. Replicas are followers that pull from the leader.

4. Topics & partitions — the unit of parallelism

• A topic is a logical stream (e.g. orders.placed).
• A topic is split into N partitions — each is an ordered, append-only log on disk.
• Within a partition, messages are strictly ordered. Across partitions, no global order.
• Partition key decides which partition a message goes to: partition = hash(key) % N.

Key choices:

• Pick a key so that ordering matters within the key (e.g. user_id). All events for one user land on one partition, processed in order.
• Don't pick a key with bad cardinality (country → hot partition for the dominant country).
• Partition count is fixed-ish — you can add but not remove cleanly. Start at 2× expected concurrent consumers.

5. The controller and metadata (KRaft replaces ZooKeeper)

The controller elects partition leaders, watches broker health, and maintains the cluster metadata. Up to Kafka 2.x this lived in ZooKeeper. From Kafka 3.3+ (production-ready 3.7+), KRaft mode replaces ZooKeeper with a self-hosted Raft quorum. Less operational surface, faster failover, official direction.

6. The on-disk log

Each partition is a series of segment files (.log + .index):

topic-partition-dir/
  00000000000000000000.log     # offsets 0–1,099,999
  00000000000000000000.index
  00000000000001100000.log     # 1.1M onward
  00000000000001100000.index

• New writes append to the active segment.
• Old segments are deleted (retention by time/size) or compacted (keep latest per key).
• The .index is a sparse offset → file-offset map, used to seek without scanning.
• Writes are sequential disk I/O — the killer feature. Kafka relies on the page cache more than its own caches.

7. Producer side — the knobs that matter

from confluent_kafka import Producer
p = Producer({
  "bootstrap.servers": "broker:9092",
  "acks": "all",                # see below
  "enable.idempotence": True,   # see below
  "compression.type": "lz4",    # 'snappy' or 'lz4' for prod
  "linger.ms": 5,               # batch window
  "batch.size": 65536,          # max bytes per batch
})
p.produce("orders.placed", key=order.user_id.encode(),
          value=json.dumps(order).encode())
p.flush()

Key settings:

• acks:
  • 0 — fire and forget. Lose messages on broker crash. Almost never.
  • 1 — leader acks. Lose data if leader dies before follower replicates. OK for telemetry where some loss is fine.
  • all — leader waits for all in-sync replicas (ISR). Standard for durable data.
• enable.idempotence = True — producer dedups retries by sequence number. Required for exactly-once. Always on in modern setups.
• Batching: linger.ms = 5–20, batch.size = 64 KB typical. Bigger batches → better throughput, slightly higher latency.
• Compression: lz4 for low CPU, snappy for compatibility. ~3–5× throughput improvement, free.
• Keys: always set if you need ordering or co-location.

8. Throughput numbers (real-ish)

On a moderate 3-broker cluster (16-core, 64 GB RAM, NVMe each):

• ~1–2 GB/s sustained ingest.
• 1–2 M small messages/sec per broker.
• p99 produce latency ~5–10 ms with acks=all + idempotence.

These aren't promises — they're what well-tuned production looks like in 2026.

9. Common producer mistakes

• No key + ordering matters — random partitioning shuffles events for the same entity.
• acks=1 for financial data — silent loss on leader failover.
• No idempotence + retries on — duplicates.
• Sync produce in a tight loop — single-threaded throughput. Use async + flush() periodically.
• Tiny batch.size — misses batching benefits.
• Producer per request in a web service — connection storm. Use one long-lived producer.

10. What's next (Session 15 — Kafka Part 2)

• Replication, ISR, unclean leader election
• Consumer groups, partition assignment, rebalances
• Exactly-once semantics (idempotence + transactions)
• Schema Registry & evolution
• Production monitoring (lag, ISR shrink, throughput)

Reading material

Books:

• Kafka: The Definitive Guide, 2nd ed. — Shapira, Palino, Sivaram, Petty (the Confluent book; chs. 1–4)
• Designing Event-Driven Systems — Ben Stopford (free O'Reilly PDF from Confluent)
• Designing Data-Intensive Applications — Kleppmann (ch. 11: Stream Processing)

Papers:

• Kafka: a Distributed Messaging System for Log Processing (Kreps et al., LinkedIn, 2011) — original Kafka paper.
• The Log: What every software engineer should know — Jay Kreps (LinkedIn essay)

Official docs:

• Apache Kafka — Documentation
• Confluent — Apache Kafka 101 course (free)
• Kafka — Producer API reference

Blog posts:

• Benchmarking Apache Kafka: 2 Million Writes per Second (LinkedIn)
• Why we built Kafka — Jay Kreps
• Kafka vs Pulsar vs Kinesis — Confluent comparison

In-depth research material

• apache/kafka — github.com/apache/kafka — ~30k ★, the source; start with core/src/main/scala/kafka/server/.
• librdkafka — github.com/confluentinc/librdkafka — C client used by everything non-JVM.
• Strimzi — Kubernetes-native Kafka operator — what most cloud installs use under the hood.
• Software Engineering Daily — Kafka episodes — multiple interviews including Jay Kreps.
• Uber Engineering — uReplicator & Kafka at Uber scale — running Kafka with 100B+ msgs/day.
• Confluent — Apache Kafka 101 course — free course with code.

Videos

• Apache Kafka 101: Brokers — Confluent · 2 min — Tim Berglund's whiteboard primer on broker responsibilities.
• Apache Kafka 101: Partitioning — Confluent · 5 min — the canonical partitioning explainer; ~170k views for a reason.
• Kafka Topics, Partitions and Offsets Explained — Stephane Maarek · 7 min — Maarek is the most-watched Kafka instructor on YouTube; clean and concise.
• Kafka Crash Course — Hands-On Project — TechWorld with Nana · 1 h 8 min — end-to-end producer + consumer + broker setup; pair with this session's lab.
• Apache Kafka in 5 minutes — Stephane Maarek · 5 min — the elevator pitch; great closer once you understand the parts.

LeetCode — Design Circular Queue

• Link: https://leetcode.com/problems/design-circular-queue/
• Difficulty: Medium
• Why this problem: Fixed array with head/tail pointers; mod arithmetic.
• Time-box: 30 minutes. Look up the editorial only after.

Post-session checklist

By the end of this session you should be able to:

• State the one-line mental model and what each word means.
• Choose a partition key for a known workload and defend it.
• Explain acks 0/1/all trade-offs.
• Turn idempotence on and explain what it actually protects.
• Tune linger.ms + batch.size + compression for throughput vs latency.
• Solve design-circular-queue — same shape as Kafka segments.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.