Testing Kafka Applications

At a Glance

What You'll Learn

• Testing pyramid for Kafka applications
• Unit testing producers and consumers with mocks
• TopologyTestDriver for Kafka Streams
• Spring @EmbeddedKafka for integration tests
• Testcontainers for realistic testing
• Contract testing with schemas
• Testing error handling and retry logic

Production Story: The Test That Saved the Release

The team had deployed to staging and everything looked good—all functional tests passed. But the integration test suite caught something:

JAVA(16 lines)
Code
Loading syntax highlighter...

The test failed. Investigation revealed that a recent configuration change had accidentally disabled retries. In production, any payment gateway hiccup would have caused immediate failures instead of graceful retries.

Mental Model: Testing Pyramid for Kafka

┌──────────────────────────────────────────────────────────────────────────┐
│                    KAFKA TESTING PYRAMID                                 │
└──────────────────────────────────────────────────────────────────────────┘

                              ┌───────────┐
                              │   E2E     │  Real Kafka cluster
                              │  Tests    │  Slow, expensive
                              │  (Few)    │  CI/CD gate
                              └─────┬─────┘
                                    │
                         ┌──────────┴──────────┐
                         │                     │
                         │   Integration       │  Embedded Kafka /
                         │   Tests             │  Testcontainers
                         │   (Some)            │  Test real behavior
                         │                     │
                         └──────────┬──────────┘
                                    │
              ┌─────────────────────┴─────────────────────┐
              │                                           │
              │            Unit Tests                     │  Mocks, stubs
              │            (Many)                         │  TopologyTestDriver
              │                                           │  Fast, isolated
              │                                           │
              └───────────────────────────────────────────┘

  WHAT TO TEST AT EACH LEVEL:

  Unit Tests:
  • Message serialization/deserialization
  • Business logic in consumers
  • Stream topology transformations
  • Error classification

  Integration Tests:
  • Producer/consumer wiring
  • Retry and error handling
  • Transaction behavior
  • Schema compatibility

  E2E Tests:
  • Full flow across services
  • Performance under load
  • Failure recovery
  • Data consistency

Deep Dive: Unit Testing Producers

Testing with MockProducer

JAVA(112 lines)
Code
Loading syntax highlighter...
Show all 112 lines

Testing Spring KafkaTemplate

JAVA(58 lines)
Code
Loading syntax highlighter...
Show all 58 lines

Deep Dive: Unit Testing Consumers

Testing Consumer Logic

JAVA(93 lines)
Code
Loading syntax highlighter...
Show all 93 lines

Deep Dive: Kafka Streams with TopologyTestDriver

Basic Topology Testing

JAVA(114 lines)
Code
Loading syntax highlighter...
Show all 114 lines

Testing Windowed Operations

JAVA(51 lines)
Code
Loading syntax highlighter...
Show all 51 lines

Testing Stream-Table Joins

JAVA(33 lines)
Code
Loading syntax highlighter...

Deep Dive: Integration Testing with @EmbeddedKafka

Basic Embedded Kafka Test

JAVA(67 lines)
Code
Loading syntax highlighter...
Show all 67 lines

Testing Error Handling

JAVA(93 lines)
Code
Loading syntax highlighter...
Show all 93 lines

Deep Dive: Testcontainers for Realistic Testing

Kafka Testcontainers Setup

JAVA(74 lines)
Code
Loading syntax highlighter...
Show all 74 lines

Full Stack with Schema Registry

JAVA(71 lines)
Code
Loading syntax highlighter...
Show all 71 lines

Deep Dive: Testing Transactions

JAVA(69 lines)
Code
Loading syntax highlighter...
Show all 69 lines

Deep Dive: Contract Testing

Producer Contract Test

JAVA(52 lines)
Code
Loading syntax highlighter...
Show all 52 lines

Consumer Contract Test

JAVA(75 lines)
Code
Loading syntax highlighter...
Show all 75 lines

Common Mistakes

1. Not Waiting for Async Operations

JAVA(9 lines)
Code
Loading syntax highlighter...

2. Shared State Between Tests

JAVA(14 lines)
Code
Loading syntax highlighter...

3. Hard-Coded Ports

JAVA(7 lines)
Code
Loading syntax highlighter...

4. Not Testing Error Paths

JAVA(12 lines)
Code
Loading syntax highlighter...

Interview Questions

Q1: "How do you test Kafka Streams topologies efficiently?"

JAVA(14 lines)
Code
Loading syntax highlighter...

• No Kafka broker needed (fast)
• Deterministic time control (advanceWallClockTime)
• Direct state store access
• Run thousands in seconds

• Real serialization issues
• Partition assignment
• Consumer group behavior
• Network failures

Those need integration tests with real Kafka."

Q2: "What's the difference between @EmbeddedKafka and Testcontainers?"

JAVA
Code
Loading syntax highlighter...

• In-process, starts fast (~2-3 seconds)
• Lighter resource usage
• Spring integration out of box
• BUT: Not exactly like production Kafka

JAVA(2 lines)
Code
Loading syntax highlighter...

• Real Docker container
• Exact production image (Confluent, etc.)
• Can add Schema Registry, Connect
• Slower startup (~10-15 seconds)
• BUT: More realistic

Embedded Kafka for:

• Unit/fast integration tests
• CI pipelines where speed matters
• Simple producer/consumer tests

Testcontainers for:

• Full integration tests
• Schema Registry testing
• Kafka Connect testing
• Production parity tests
• Security configuration tests"

Q3: "How do you test exactly-once semantics in Kafka?"

JAVA(5 lines)
Code
Loading syntax highlighter...

JAVA(5 lines)
Code
Loading syntax highlighter...

JAVA(2 lines)
Code
Loading syntax highlighter...

PROPERTIES(3 lines)
Code
Loading syntax highlighter...

• Committed messages are visible
• Rolled back messages are not visible
• Consumer offsets commit with transaction
• Idempotent producer behavior (enable.idempotence)"

Q4: "How do you ensure your tests don't interfere with each other?"

JAVA(2 lines)
Code
Loading syntax highlighter...

JAVA(2 lines)
Code
Loading syntax highlighter...

JAVA(5 lines)
Code
Loading syntax highlighter...

JAVA(2 lines)
Code
Loading syntax highlighter...

JAVA(2 lines)
Code
Loading syntax highlighter...

• Use unique topics for parallel test execution
• Use unique groups for consumer tests
• Clean database state before each test
• Avoid @DirtiesContext if possible (slow)"

Q5: "How do you test error handling and retry logic?"

JAVA(5 lines)
Code
Loading syntax highlighter...

JAVA(15 lines)
Code
Loading syntax highlighter...

JAVA(13 lines)
Code
Loading syntax highlighter...

JAVA(8 lines)
Code
Loading syntax highlighter...

The key is testing both the happy path AND all failure modes."

Summary & Key Takeaways

Testing Strategy Summary

┌──────────────────────────────────────────────────────────────────────────┐
│                    KAFKA TESTING SUMMARY                                 │
├────────────────────────────────┬─────────────────────────────────────────┤
│ Test Type                      │ Tool/Approach                           │
├────────────────────────────────┼─────────────────────────────────────────┤
│ Unit - Producer logic          │ MockProducer, Mockito                   │
│ Unit - Consumer logic          │ Direct method calls, mocks              │
│ Unit - Streams topology        │ TopologyTestDriver                      │
│ Integration - Basic            │ @EmbeddedKafka                          │
│ Integration - Realistic        │ Testcontainers                          │
│ Integration - With SR          │ Testcontainers + Schema Registry        │
│ Contract - Producer            │ Verify message structure                │
│ Contract - Consumer            │ Test with minimal/maximal messages      │
│ E2E                            │ Full environment, real Kafka            │
└────────────────────────────────┴─────────────────────────────────────────┘

Essential Takeaways

1. TopologyTestDriver is your friend - Fast, deterministic Streams testing without a broker.
2. @EmbeddedKafka for speed - Quick integration tests, good enough for most cases.
3. Testcontainers for realism - When you need production parity.
4. Always test error paths - Retry, DLQ, and deserialization failures.
5. Isolate your tests - Unique topics/groups, cleanup between tests.
6. Wait for async operations - Use await() or CompletableFuture.get().

Quick Reference

┌──────────────────────────────────────────────────────────────────────────┐
│                    TESTING QUICK REFERENCE                               │
├────────────────────────────────┬─────────────────────────────────────────┤
│ Tool                           │ Usage                                   │
├────────────────────────────────┼─────────────────────────────────────────┤
│ MockProducer                   │ new MockProducer<>(autoComplete, ...)   │
│ TopologyTestDriver             │ new TopologyTestDriver(topology, props) │
│ @EmbeddedKafka                 │ @EmbeddedKafka(topics = {...})          │
│ KafkaContainer                 │ new KafkaContainer(image)               │
│ KafkaTestUtils                 │ .getRecords(), .getSingleRecord()       │
│ await()                        │ await().atMost(10, SECONDS).until(...)  │
├────────────────────────────────┼─────────────────────────────────────────┤
│ Common Assertions              │                                         │
├────────────────────────────────┼─────────────────────────────────────────┤
│ Verify message sent            │ mockProducer.history()                  │
│ Verify output topic            │ outputTopic.readKeyValue()              │
│ Verify state store             │ testDriver.getKeyValueStore()           │
│ Verify processing              │ await() + repository check              │
└────────────────────────────────┴─────────────────────────────────────────┘

---

Series Navigation