Devops
Partitions & Replication
📋 At a Glance
| Aspect | Details |
|---|---|
| Difficulty | 🟡 Intermediate |
| Prerequisites | Part 1 (Architecture & Storage) |
| Key Concepts | Partitions, replication factor, rack awareness, reassignment |
| Time Investment | 30 minutes read + 45 minutes practice |
| Payoff | Design topics that scale and survive failures |
🎯 What You'll Learn
After this article, you'll be able to:
- Choose the right partition count for your workload
- Understand replication and how data is distributed across brokers
- Configure rack awareness for datacenter fault tolerance
- Perform partition reassignment safely in production
- Diagnose partition imbalance and hot partition issues
🔥 Production Story: The Hot Partition Nightmare
The Setup: An order processing system with topic
orders (10 partitions, 3x replication). During a Black Friday sale, the system collapsed.The Symptoms:
Consumer lag: Partition 0: 50 messages Partition 1: 45 messages Partition 2: 52 messages Partition 3: 2,500,000 messages ← WTF! Partition 4: 48 messages ...
One partition had 50,000x the lag of others!
The Investigation:
JAVA(2 lines)CodeLoading syntax highlighter...
Looks reasonable—key by customer ID. But wait...
SQL(11 lines)CodeLoading syntax highlighter...
The Root Cause: Guest checkout used
customer_id = "GUEST" for all anonymous users. Since partitioning is hash(key) % partitions, all 2.5 million guest orders went to the same partition!The Fix:
JAVA(6 lines)CodeLoading syntax highlighter...
Lesson Learned: Partition key design is critical. A single hot key can bottleneck your entire system.
🧠 Mental Model: Partitions and Replication
┌─────────────────────────────────────────────────────────────────┐ │ TOPIC: orders (6 partitions, RF=3) │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Broker 1 Broker 2 Broker 3 │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ P0 (L) │ │ P0 (F) │ │ P0 (F) │ │ │ │ P1 (F) │ │ P1 (L) │ │ P1 (F) │ │ │ │ P2 (F) │ │ P2 (F) │ │ P2 (L) │ │ │ │ P3 (L) │ │ P3 (F) │ │ P3 (F) │ │ │ │ P4 (F) │ │ P4 (L) │ │ P4 (F) │ │ │ │ P5 (F) │ │ P5 (F) │ │ P5 (L) │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ L = Leader (handles reads/writes) │ │ F = Follower (replicates from leader) │ │ │ │ ───────────────────────────────────────────────────────────── │ │ │ │ PRODUCER PERSPECTIVE: │ │ │ │ Producer │ │ │ │ │ │ Message with key="user123" │ │ │ │ │ ▼ │ │ Partitioner: hash("user123") % 6 = 2 │ │ │ │ │ ▼ │ │ Send to Broker 3 (leader of P2) │ │ │ │ ───────────────────────────────────────────────────────────── │ │ │ │ CONSUMER PERSPECTIVE: │ │ │ │ Consumer Group: order-processors (3 consumers) │ │ │ │ Consumer 1 Consumer 2 Consumer 3 │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ P0, P1 │ │ P2, P3 │ │ P4, P5 │ │ │ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ Each partition assigned to exactly one consumer in group │ │ │ └─────────────────────────────────────────────────────────────────┘
🔬 Deep Dive
1. Understanding Partitions
A partition is the unit of:
- Parallelism: More partitions = more consumers can process in parallel
- Ordering: Messages within a partition are strictly ordered
- Storage: Each partition is a separate log on disk
Partition assignment algorithm (default):
JAVA(14 lines)CodeLoading syntax highlighter...
Ordering guarantees:
┌─────────────────────────────────────────────────────────────────┐ │ ORDERING GUARANTEES │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ WITHIN A PARTITION: Total ordering guaranteed │ │ │ │ Partition 0: [M1] → [M2] → [M3] → [M4] │ │ Always consumed in this order │ │ │ │ ───────────────────────────────────────────────────────────── │ │ │ │ ACROSS PARTITIONS: No ordering guarantee │ │ │ │ P0: [M1] → [M3] │ │ P1: [M2] → [M4] │ │ │ │ Consumer might see: M2, M1, M4, M3 (any interleaving) │ │ │ │ ──────────────────────────────────────────────────────────── │ │ │ │ IMPLICATION: If you need order, use the same key! │ │ │ │ // All orders for user123 go to same partition │ │ kafkaTemplate.send("orders", "user123", orderEvent); │ │ │ └─────────────────────────────────────────────────────────────────┘
2. How Many Partitions?
The formula (starting point):
Target Partitions = max( Throughput / Max Throughput per Partition, Consumer Count in Largest Consumer Group )
Practical guidelines:
| Factor | Guidance |
|---|---|
| Throughput | ~10 MB/s or 10K msg/s per partition (varies by message size) |
| Consumers | At least as many partitions as max consumers |
| Over-partitioning | More partitions = more overhead, but easier to scale later |
| Rule of thumb | Start with # brokers × 10 for high-throughput topics |
Example calculation:
Requirements: - Expected throughput: 100K messages/sec - Message size: 1KB - Max consumers in any group: 20 Calculations: - Data throughput: 100K × 1KB = 100 MB/s - Partitions for throughput: 100 MB/s ÷ 10 MB/s = 10 - Partitions for consumers: 20 Result: max(10, 20) = 20 partitions minimum Recommendation: 24-30 partitions (room to grow)
Why not just use 1000 partitions?
┌─────────────────────────────────────────────────────────────────┐ │ PARTITION COUNT TRADE-OFFS │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ TOO FEW PARTITIONS TOO MANY PARTITIONS │ │ ───────────────── ────────────────── │ │ • Limited parallelism • More file handles │ │ • Can't add more consumers • More memory per broker │ │ • Hot partitions more likely • Longer leader election │ │ • Hard to scale later • Slower rebalancing │ │ • More ZK/KRaft metadata │ │ │ │ SWEET SPOT │ │ ────────── │ │ • 2-3x expected consumer count │ │ • Allows for growth │ │ • Manageable overhead │ │ • Typically: dozens to low hundreds per topic │ │ │ └─────────────────────────────────────────────────────────────────┘
Can you change partition count later?
BASH(10 lines)CodeLoading syntax highlighter...
3. Replication Deep Dive
Replication factor determines how many copies of each partition exist:
┌─────────────────────────────────────────────────────────────────┐ │ REPLICATION MECHANICS │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Topic: payments, Partition 0, RF=3 │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Broker 1 (Leader) │ │ │ │ ┌─────────────────────────────────────────────────────┐│ │ │ │ │ Offset: 0 1 2 3 4 5 6 7 ││ │ │ │ │ [M0] [M1] [M2] [M3] [M4] [M5] [M6] [M7] ││ │ │ │ │ ↑ ││ │ │ │ │ High Watermark (HW) ││ │ │ │ │ (committed, safe) ││ │ │ │ └─────────────────────────────────────────────────────┘│ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ │ Fetch requests │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Broker 2 (Follower, ISR) │ │ │ │ ┌─────────────────────────────────────────────────────┐│ │ │ │ │ Offset: 0 1 2 3 4 5 6 7 ││ │ │ │ │ [M0] [M1] [M2] [M3] [M4] [M5] [M6] [M7] ││ │ │ │ │ ↑ ││ │ │ │ │ Caught up! ││ │ │ │ └─────────────────────────────────────────────────────┘│ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Broker 3 (Follower, ISR) │ │ │ │ ┌─────────────────────────────────────────────────────┐│ │ │ │ │ Offset: 0 1 2 3 4 5 6 ││ │ │ │ │ [M0] [M1] [M2] [M3] [M4] [M5] [M6] ││ │ │ │ │ ↑ ││ │ │ │ │ 1 message behind ││ │ │ │ │ (still in ISR) ││ │ │ │ └─────────────────────────────────────────────────────┘│ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ ISR (In-Sync Replicas) = {Broker1, Broker2, Broker3} │ │ All replicas within replica.lag.time.max.ms are "in sync" │ │ │ └─────────────────────────────────────────────────────────────────┘
Replication factor recommendations:
| Environment | RF | Reason |
|---|---|---|
| Development | 1 | No redundancy needed |
| Production (standard) | 3 | Survives 2 broker failures |
| Critical data | 3-5 | Extra safety margin |
Key configuration:
PROPERTIES(8 lines)CodeLoading syntax highlighter...
4. Rack Awareness
Without rack awareness, all replicas might be on the same rack:
┌─────────────────────────────────────────────────────────────────┐ │ WITHOUT RACK AWARENESS │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Rack A Rack B │ │ ┌─────────────────────────┐ ┌─────────────────────────┐ │ │ │ Broker 1 │ │ Broker 3 │ │ │ │ ├── P0 (Leader) │ │ (no replicas) │ │ │ │ └── P0 (Follower) │ │ │ │ │ ├─────────────────────────┤ ├─────────────────────────┤ │ │ │ Broker 2 │ │ Broker 4 │ │ │ │ └── P0 (Follower) │ │ (no replicas) │ │ │ └─────────────────────────┘ └─────────────────────────┘ │ │ │ │ ⚠️ If Rack A loses power, ALL replicas of P0 are lost! │ │ │ └─────────────────────────────────────────────────────────────────┘
With rack awareness:
┌─────────────────────────────────────────────────────────────────┐ │ WITH RACK AWARENESS │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Rack A Rack B │ │ ┌─────────────────────────┐ ┌─────────────────────────┐ │ │ │ Broker 1 │ │ Broker 3 │ │ │ │ └── P0 (Leader) │ │ └── P0 (Follower) │ │ │ ├─────────────────────────┤ ├─────────────────────────┤ │ │ │ Broker 2 │ │ Broker 4 │ │ │ │ └── P0 (Follower) │ │ (balanced elsewhere) │ │ │ └─────────────────────────┘ └─────────────────────────┘ │ │ │ │ ✓ Replicas spread across racks │ │ ✓ Rack A failure: P0 still available from Rack B │ │ │ └─────────────────────────────────────────────────────────────────┘
Configuration:
PROPERTIES(5 lines)CodeLoading syntax highlighter...
Creating rack-aware topics:
BASH(6 lines)CodeLoading syntax highlighter...
5. Partition Reassignment
When you add brokers or rebalance load, you need to move partitions:
BASH(28 lines)CodeLoading syntax highlighter...
Throttling reassignment (crucial for production):
BASH(13 lines)CodeLoading syntax highlighter...
6. Spring Kafka: Partition Configuration
Creating topics programmatically:
JAVA(20 lines)CodeLoading syntax highlighter...
Custom partitioner:
JAVA(42 lines)CodeLoading syntax highlighter...
Sending to specific partition:
JAVA(20 lines)CodeLoading syntax highlighter...
7. Diagnosing Partition Issues
Check partition distribution:
BASH(13 lines)CodeLoading syntax highlighter...
Check consumer lag per partition:
BASH(8 lines)CodeLoading syntax highlighter...
Diagnose hot partition:
JAVA(25 lines)CodeLoading syntax highlighter...
⚠️ Common Mistakes
Mistake 1: Using High-Cardinality Keys Incorrectly
JAVA(12 lines)CodeLoading syntax highlighter...
Mistake 2: Wrong Partition Count
JAVA(14 lines)CodeLoading syntax highlighter...
Mistake 3: Replication Factor = 1 in Production
PROPERTIES(8 lines)CodeLoading syntax highlighter...
Mistake 4: Increasing Partitions Without Understanding Impact
BASH(8 lines)CodeLoading syntax highlighter...
Mistake 5: Not Throttling Reassignment
BASH(9 lines)CodeLoading syntax highlighter...
🐛 Debug This
You've deployed 6 consumers for a topic with 12 partitions. Expected: 2 partitions each. Actual:
Consumer 1: P0, P1, P2, P3, P4, P5 Consumer 2: P6, P7, P8, P9, P10, P11 Consumer 3: (none) Consumer 4: (none) Consumer 5: (none) Consumer 6: (none)
Only 2 consumers are getting partitions. What's wrong?
Click to reveal analysis
Possible causes:
-
Different group IDs: Consumers might be in different groupsBASH(3 lines)CodeLoading syntax highlighter...
-
Static membership mismatch: If using static membership, instance IDs might conflictJAVA(2 lines)CodeLoading syntax highlighter...
-
Partition assignment strategy: RangeAssignor can cause uneven distributionJAVA(7 lines)CodeLoading syntax highlighter...
-
Consumer not fully started: Some consumers might still be starting upBASH(3 lines)CodeLoading syntax highlighter...
Most likely: Different
group.id values. Verify all consumers use the exact same group ID:JAVA(2 lines)CodeLoading syntax highlighter...
💻 Exercises
Exercise 1: Partition Distribution Analysis
Create a topic with 10 partitions. Send 10,000 messages with various key patterns and analyze distribution:
JAVA(6 lines)CodeLoading syntax highlighter...
Exercise 2: Hot Partition Simulation
Create a scenario with a hot partition and measure impact:
JAVA(4 lines)CodeLoading syntax highlighter...
Exercise 3: Rack Awareness Setup
Configure a 4-broker local cluster with rack awareness:
BASH(3 lines)CodeLoading syntax highlighter...
Exercise 4: Partition Reassignment
Practice safe reassignment:
BASH(5 lines)CodeLoading syntax highlighter...
Exercise 5: Custom Partitioner
Implement a partitioner that:
- Routes "priority" orders to partition 0
- Routes orders by region (EMEA → P1-3, APAC → P4-6, AMER → P7-9)
- Falls back to hash for unknown regions
🎤 Interview Questions
Q1: How do you decide the number of partitions for a new topic?
Answer: Consider these factors:
-
Consumer parallelism: Partitions ≥ max consumers in any consumer group
If max consumers = 20, need at least 20 partitions -
Throughput requirements: ~10 MB/s per partition (varies)
100 MB/s needed → at least 10 partitions -
Ordering requirements: More partitions = less ordering
If you need all events ordered, use 1 partition (limits throughput) -
Growth: Increasing partitions later breaks key-based ordering
Better to over-provision initially: target × 2
Formula:
partitions = max( target_throughput / throughput_per_partition, max_consumers ) × growth_factor
Example: 50 MB/s, 10 consumers, 2x growth factor
max(5, 10) × 2 = 20 partitions
Q2: What happens when you add partitions to an existing topic?
Answer: Adding partitions has these effects:
-
New messages with existing keys may go to different partitions:
Before: hash("user123") % 10 = 3 After: hash("user123") % 15 = 8Messages for the same key are now split across partitions.
-
Ordering guarantee is broken for existing keys: New messages for "user123" go to partition 8, but old messages are still in partition 3.
-
Consumers need rebalance: Consumer group will rebalance to assign new partitions.
-
Existing data stays in place: Old messages don't move.
When it's safe to add partitions:
- Topic doesn't use key-based partitioning
- Application doesn't require ordering
- You can tolerate a period of unordered processing
Alternative: Create new topic, migrate consumers, re-key if needed.
Q3: Explain the relationship between partitions, replication factor, and ISR.
Answer:
Partitions: Independent ordered logs within a topic. Provide parallelism.
Replication Factor (RF): How many copies of each partition exist.
RF=3 means each partition has 3 replicas (1 leader + 2 followers)
ISR (In-Sync Replicas): Subset of replicas that are caught up with the leader.
If RF=3 and one follower is slow: ISR={leader, follower1} (size 2)
How they interact:
min.insync.replicas=2, acks=all: - Producer waits for leader + 1 follower to acknowledge - If ISR drops to 1, producers get NotEnoughReplicas error - Data is durable on 2+ machines before ack
Example scenario:
RF=3, min.insync.replicas=2 Normal: ISR={1,2,3} → Writes succeed (3 ≥ 2) 1 broker down: ISR={1,2} → Writes succeed (2 ≥ 2) 2 brokers down: ISR={1} → Writes FAIL (1 < 2)
Q4: How does rack awareness improve fault tolerance?
Answer: Rack awareness ensures replicas are spread across failure domains:
Without rack awareness:
Rack A: [P0-leader, P0-follower, P0-follower] Rack B: [empty] Rack A power failure = ALL replicas lost = DATA LOSS
With rack awareness:
Rack A: [P0-leader, P0-follower] Rack B: [P0-follower] Rack A power failure = P0-follower in Rack B becomes leader = NO DATA LOSS
Configuration:
PROPERTIES(2 lines)CodeLoading syntax highlighter...
Best practices:
- RF should span at least 2 racks
- For RF=3, use 3 racks if possible
- Align with cloud availability zones
Q5: A consumer group has 6 consumers but some partitions are unassigned. What could be wrong?
Answer: Several possibilities:
-
Partition count < consumer count:
Topic has 4 partitions, 6 consumers Result: 2 consumers idle (partitions can't be shared) -
Different group IDs:JAVA(2 lines)CodeLoading syntax highlighter...
-
Static membership collision:JAVA(2 lines)CodeLoading syntax highlighter...
-
Assignment strategy issue: RangeAssignor with multiple topics can leave consumers idle.
-
Consumer failed health check: Consumer might be considered dead.
Diagnosis:
BASH(5 lines)CodeLoading syntax highlighter...
Most common cause: Misconfigured
group.id (not actually the same group).📝 Summary & Key Takeaways
Partition Principles
- Unit of parallelism, ordering, and storage
- Same key → same partition → ordering guaranteed
- More partitions = more consumers possible
Sizing Guidelines
partitions = max(throughput_needs, consumer_count) × growth_factor
Replication
- RF=3 for production
- ISR tracks healthy replicas
min.insync.replicasdefines minimum for writes
Key Gotchas
- Adding partitions breaks key-based ordering
- Hot keys create hot partitions
- RF=1 means any broker failure loses data
📋 Quick Reference
BASH(20 lines)CodeLoading syntax highlighter...
📅 Review Schedule
- Day 1: Understand partition-key relationship
- Day 3: Practice calculating partition counts
- Day 7: Experiment with reassignment
- Day 14: Review rack awareness
- Day 30: Diagnose partition issues without notes
📚 Series Navigation
- Previous: Part 1 - Architecture & Storage Engine
- Next: Part 3 - Leaders, ISR & Fault Tolerance
- Index: Kafka Compendium Series