How to Use This Series

📋 Who This Series Is For

This series is for mid-senior developers who:

✅ Already use Kafka in production
✅ Can write producers and consumers
✅ Know what topics, partitions, and consumer groups are
✅ Want to understand why things work the way they do
✅ Need to debug production issues confidently
✅ Are preparing for senior-level interviews

This series is NOT for:

❌ Complete beginners (start with Kafka Quickstart first)
❌ People who just need to send a message (use the docs)
❌ Those looking for copy-paste solutions without understanding

🎯 What You'll Master

By the end of this series, you'll be able to:

Skill	Parts
Explain why Kafka is fast	1-2
Configure producers for your exact guarantees	5-7
Debug consumer lag and rebalancing issues	8-10
Implement exactly-once processing	11
Design schemas that evolve safely	12
Secure a multi-tenant cluster	13
Build streaming applications	15-17
Design event-driven architectures	18
Handle errors gracefully	19
Pass senior Kafka interviews	All

🗺️ Series Roadmap

┌─────────────────────────────────────────────────────────────────┐
│                    KAFKA COMPENDIUM ROADMAP                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   FUNDAMENTALS (Parts 1-4)          ← Start here                │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ 1. Architecture & Storage    Why Kafka is fast          │   │
│   │ 2. Partitions & Replication  Data distribution          │   │
│   │ 3. Leaders, ISR & Faults     Reliability guarantees     │   │
│   │ 4. KRaft vs ZooKeeper        Cluster coordination       │   │
│   └─────────────────────────────────────────────────────────┘   │
│                           ↓                                     │
│   PRODUCERS (Parts 5-7)                                         │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ 5. Producer Internals        How sending works          │   │
│   │ 6. Delivery Guarantees       acks, idempotence          │   │
│   │ 7. Advanced Patterns         Transactions, ordering     │   │
│   └─────────────────────────────────────────────────────────┘   │
│                           ↓                                     │
│   CONSUMERS (Parts 8-11)                                        │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ 8. Consumer Internals        How fetching works         │   │
│   │ 9. Groups & Rebalancing      Coordination patterns      │   │
│   │ 10. Offset Management        Commit strategies          │   │
│   │ 11. Exactly-Once             End-to-end guarantees      │   │
│   └─────────────────────────────────────────────────────────┘   │
│                           ↓                                     │
│   OPERATIONS (Parts 12-14)                                      │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ 12. Schema Registry          Schema evolution           │   │
│   │ 13. Security                 AuthN, AuthZ, encryption   │   │
│   │ 14. Monitoring               Metrics, alerting, ops     │   │
│   └─────────────────────────────────────────────────────────┘   │
│                           ↓                                     │
│   KAFKA STREAMS (Parts 15-17)                                   │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ 15. Streams Fundamentals     KStream, KTable, topology  │   │
│   │ 16. State Stores             Stateful processing        │   │
│   │ 17. Windowing & Joins        Time-based operations      │   │
│   └─────────────────────────────────────────────────────────┘   │
│                           ↓                                     │
│   PATTERNS (Parts 18-20)                                        │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ 18. Event-Driven Patterns    Sourcing, CQRS, Saga       │   │
│   │ 19. Error Handling           DLQ, retries, recovery     │   │
│   │ 20. Testing                  Unit, integration, E2E     │   │
│   └─────────────────────────────────────────────────────────┘   │
│                           ↓                                     │
│   REFERENCE (Part 21)                                           │
│   ┌─────────────────────────────────────────────────────────┐   │
│   │ 21. Cheatsheet               Commands, configs, trees   │   │
│   └─────────────────────────────────────────────────────────┘   │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

📚 Learning Paths

Path 1: Full Deep Dive (Recommended)

Time: ~35 hours | For: Comprehensive understanding

Read sequentially: Parts 0 → 21

Path 2: Producer Focus

Time: ~8 hours | For: Backend developers sending events

Part 1: Architecture (why it matters)
Parts 5-7: Producer deep dive
Part 12: Schema Registry
Part 19: Error handling

Path 3: Consumer Focus

Time: ~10 hours | For: Event processors, workers

Part 1: Architecture
Parts 8-11: Consumer deep dive
Part 14: Monitoring (lag!)
Part 19: Error handling

Path 4: Kafka Streams

Time: ~12 hours | For: Stream processing engineers

Parts 1-2: Fundamentals
Part 10: Offset management
Parts 15-17: Kafka Streams
Part 20: Testing

Path 5: Interview Prep

Time: ~6 hours | For: Senior role interviews

Focus on Interview Questions sections in:

Part 3: Fault tolerance
Part 6: Delivery guarantees
Part 9: Consumer groups
Part 11: Exactly-once
Part 21: Cheatsheet

🛠️ Prerequisites Check

Before starting, you should be able to answer:

Question	If No, Study
What is a Kafka topic?	Kafka Quickstart
What is a partition?	Kafka Quickstart
What is a consumer group?	Kafka Basics
Can you write a Spring Boot app?	Spring Boot guides
What is a distributed system?	Distributed Systems basics

Quick self-test: If you can explain "consumer group rebalancing" in one sentence, you're ready.

💻 Local Environment Setup

You'll need a local Kafka cluster to practice. Here's a production-like setup using KRaft (no ZooKeeper):

Docker Compose Setup

YAML(81 lines)
Code
Loading syntax highlighter...

Starting the Environment

BASH(14 lines)
Code
Loading syntax highlighter...

Spring Boot Dependencies

XML(40 lines)
Code
Loading syntax highlighter...

YAML(14 lines)
Code
Loading syntax highlighter...

📖 Article Structure

Every article in this series follows the same structure:

1. At a Glance

Quick reference table with difficulty, prerequisites, and time investment.

2. What You'll Learn

4-5 concrete learning objectives.

3. Production Story 🔥

A real-world scenario where things went wrong. Each story includes:

The setup (what the team was trying to do)
The symptoms (what went wrong)
The investigation (how they found the problem)
The root cause (the actual issue)
The fix (how they solved it)

Why? These stories make abstract concepts memorable.

4. Mental Model 🧠

ASCII diagram showing how the concept works visually. These diagrams are designed to be the "picture in your head" when thinking about the concept.

5. Deep Dive 🔬

The main technical content, typically 5-8 sections with code examples.

6. Common Mistakes ⚠️

5 mistakes with "wrong" and "right" examples. Learn from others' errors.

7. Debug This 🐛

An interactive scenario: given symptoms, figure out what's wrong. Answers are in a collapsible section.

8. Exercises 💻

5 hands-on exercises ranging from basic to advanced.

9. Interview Questions 🎤

5 senior-level questions with detailed answers. These are questions you might face in staff/principal engineer interviews.

10. Summary & Quick Reference

Key takeaways and copy-paste reference snippets.

🎯 How to Study

For Maximum Retention

Read the Production Story first - It gives context for why the concept matters
Draw the Mental Model yourself - Don't just look at it, recreate it
Run the code examples - Type them, don't copy-paste
Do at least 2 exercises - Application beats passive reading
Explain Interview Questions out loud - Teaching solidifies learning

Spaced Repetition Schedule

For each part:

Day 1: Read and do exercises
Day 3: Review mental model, redo Debug This
Day 7: Answer interview questions without looking
Day 14: Quick review of summary
Day 30: Full review if preparing for interviews

⚡ Quick Start Verification

Let's verify your environment works. Create a simple producer/consumer:

JAVA(24 lines)
Code
Loading syntax highlighter...

JAVA(9 lines)
Code
Loading syntax highlighter...

JAVA(18 lines)
Code
Loading syntax highlighter...

If you see messages being sent and received, you're ready!

🔑 Key Conventions

Configuration Notation

JAVA(4 lines)
Code
Loading syntax highlighter...

Code Style

All examples use Java 21+ features
All Spring examples use Spring Boot 3.x and Spring Kafka 3.x
Kafka version is 3.6+ (KRaft mode)
Examples are complete and runnable (not snippets)

Terminology

Term	Meaning
Broker	Kafka server instance
Leader	Broker responsible for partition reads/writes
Follower	Broker that replicates from leader
ISR	In-Sync Replicas - followers caught up with leader
Consumer Group	Set of consumers sharing topic consumption
Offset	Position of a message in a partition
Lag	How far behind a consumer is
Rebalance	Redistributing partitions among consumers

🆘 Getting Help

If you get stuck:

Check the Debug This section - Your issue might be a common one
Review the Common Mistakes - You might be hitting a known pitfall
Use Kafka UI - Visual inspection often reveals the problem
Check broker logs - docker compose logs kafka

📚 What's Next?

Ready to dive in? Start with Part 1: Architecture & Storage Engine to understand why Kafka is fundamentally different from other message brokers.

You'll learn:

Why Kafka uses a log-structured storage
How zero-copy transfer makes it fast
What the page cache does for performance
How messages are actually stored on disk

📅 Review Schedule

Now: Set up your local environment
Part 1: Understand why Kafka is fast
Part 5: Send your first production-ready message
Part 8: Consume with confidence
Part 21: Have a complete mental model of Kafka

Next: Part 1 - Architecture & Storage Engine
Index: Kafka Compendium Series