Devops
Security (AuthN, AuthZ, Encryption)
At a Glance
| Aspect | Details |
|---|---|
| Topic | Authentication (SASL), Authorization (ACLs), Encryption (TLS) |
| Complexity | Intermediate-Advanced |
| Prerequisites | Parts 1-4 (Fundamentals) |
| Time | 90 minutes |
| Spring Kafka | SSL/SASL configuration, security properties |
What You'll Learn
After completing this article, you will be able to:
- Configure authentication using SASL/PLAIN, SASL/SCRAM, and mTLS
- Implement authorization with Kafka ACLs
- Enable encryption in transit with SSL/TLS
- Set up Spring Kafka clients with security configurations
- Design multi-tenant Kafka clusters with proper isolation
Production Story: The Unauthorized Data Access
The Incident
Our company ran a shared Kafka cluster for multiple teams. One Monday, the data governance team discovered that the marketing analytics service had been consuming from the
customer-pii topic - a topic containing sensitive personal information that marketing should never have access to.The Investigation
BASH(8 lines)CodeLoading syntax highlighter...
The audit showed:
┌─────────────────────────────────────────────────────────────────────┐ │ THE UNAUTHORIZED ACCESS │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ Cluster Configuration: │ │ • No authentication (PLAINTEXT listeners) │ │ • No authorization (ACLs disabled) │ │ • No encryption (data visible on network) │ │ │ │ What happened: │ │ 1. Marketing developer needed customer IDs │ │ 2. Found 'customer-pii' topic while exploring │ │ 3. Started consuming - no access check! │ │ 4. PII data flowing to marketing systems │ │ 5. Potential GDPR violation │ │ │ │ Topics in cluster: │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ customer-pii ← Sensitive! Marketing accessed this │ │ │ │ customer-orders ← Marketing should access this │ │ │ │ inventory │ │ │ │ analytics-events │ │ │ │ marketing-campaigns │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ Network visibility: │ │ Anyone on the network could see plaintext Kafka traffic │ │ Including customer names, emails, addresses... │ │ │ └─────────────────────────────────────────────────────────────────────┘
The Root Cause
"It's an internal cluster, we trust our developers" - this assumption failed:
- No authentication = anyone can connect
- No authorization = anyone can access any topic
- No encryption = data visible on network
The Fix
PROPERTIES(21 lines)CodeLoading syntax highlighter...
BASH(29 lines)CodeLoading syntax highlighter...
After the fix: Marketing service denied access to
customer-pii, audit trail for all access attempts.Mental Model: Kafka Security Layers
┌─────────────────────────────────────────────────────────────────────────┐ │ KAFKA SECURITY LAYERS │ ├─────────────────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ 1. ENCRYPTION (TLS/SSL) │ │ │ │ │ │ │ │ Protects data in transit │ │ │ │ │ │ │ │ Client ════════════════════════════════════════════► Broker │ │ │ │ Encrypted channel (TLS 1.2/1.3) │ │ │ │ │ │ │ │ Without TLS: "password123" visible on network │ │ │ │ With TLS: 0x7f3a9b2c... (encrypted) │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ 2. AUTHENTICATION (SASL/mTLS) │ │ │ │ │ │ │ │ Verifies identity: "Who are you?" │ │ │ │ │ │ │ │ ┌─────────────┐ Credentials ┌─────────────┐ │ │ │ │ │ Client │ ─────────────────►│ Broker │ │ │ │ │ │ user: app1 │ │ Verify ID │ │ │ │ │ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ │ Methods: │ │ │ │ • SASL/PLAIN - username/password (simple) │ │ │ │ • SASL/SCRAM - salted challenge-response (secure) │ │ │ │ • SASL/GSSAPI - Kerberos (enterprise) │ │ │ │ • SASL/OAUTHBEARER - OAuth2 tokens │ │ │ │ • mTLS - mutual TLS certificates │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ 3. AUTHORIZATION (ACLs) │ │ │ │ │ │ │ │ Controls access: "What can you do?" │ │ │ │ │ │ │ │ User: marketing-service │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ │ │ Resource │ Operation │ Permission │ │ │ │ │ ├───────────────────┼───────────┼─────────────────────────┤ │ │ │ │ │ topic:orders │ Read │ ALLOW │ │ │ │ │ │ topic:orders │ Write │ DENY │ │ │ │ │ │ topic:customer-* │ All │ DENY │ │ │ │ │ │ group:marketing-* │ All │ ALLOW │ │ │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ All three layers should be enabled for production security! │ │ │ └─────────────────────────────────────────────────────────────────────────┘
Deep Dive
1. Authentication Methods
SASL/PLAIN (Simple but requires TLS)
PROPERTIES(15 lines)CodeLoading syntax highlighter...
JAVA(28 lines)CodeLoading syntax highlighter...
SASL/SCRAM (More secure)
BASH(8 lines)CodeLoading syntax highlighter...
PROPERTIES(8 lines)CodeLoading syntax highlighter...
JAVA(32 lines)CodeLoading syntax highlighter...
mTLS (Mutual TLS - Certificate-based)
BASH(26 lines)CodeLoading syntax highlighter...
PROPERTIES(16 lines)CodeLoading syntax highlighter...
JAVA(24 lines)CodeLoading syntax highlighter...
2. Authorization with ACLs
BASH(43 lines)CodeLoading syntax highlighter...
ACL Operations Reference
┌─────────────────────────────────────────────────────────────────────┐ │ KAFKA ACL OPERATIONS │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ TOPIC OPERATIONS: │ │ ───────────────── │ │ • Read - Consume from topic │ │ • Write - Produce to topic │ │ • Create - Create topic (if auto.create enabled) │ │ • Delete - Delete topic │ │ • Alter - Modify topic configuration │ │ • Describe - View topic metadata │ │ • All - All of the above │ │ │ │ GROUP OPERATIONS: │ │ ───────────────── │ │ • Read - Join consumer group │ │ • Describe - View group offsets │ │ • Delete - Delete group │ │ • All - All of the above │ │ │ │ CLUSTER OPERATIONS: │ │ ────────────────── │ │ • Create - Create topics │ │ • Alter - Change cluster config │ │ • Describe - View cluster metadata │ │ • ClusterAction - Inter-broker communication │ │ • All - All of the above │ │ │ │ TRANSACTIONAL ID OPERATIONS: │ │ ───────────────────────────── │ │ • Write - Use transactional ID │ │ • Describe - View transaction state │ │ │ └─────────────────────────────────────────────────────────────────────┘
3. Complete Spring Kafka Security Configuration
JAVA(94 lines)CodeLoading syntax highlighter...
Application Properties
YAML(12 lines)CodeLoading syntax highlighter...
4. Multi-Tenant Security Design
┌─────────────────────────────────────────────────────────────────────┐ │ MULTI-TENANT KAFKA DESIGN │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ Option 1: Topic Prefix per Tenant │ │ ───────────────────────────────── │ │ │ │ Topics: │ │ ├── tenant-a.orders │ │ ├── tenant-a.users │ │ ├── tenant-b.orders │ │ └── tenant-b.users │ │ │ │ ACLs: │ │ User:tenant-a-app → topic:tenant-a.* (prefixed) → ALLOW │ │ User:tenant-b-app → topic:tenant-b.* (prefixed) → ALLOW │ │ │ │ Pros: Simple, clear isolation │ │ Cons: Topic proliferation │ │ │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ Option 2: Dedicated Clusters per Tenant │ │ ──────────────────────────────────────── │ │ │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │ Cluster-A │ │ Cluster-B │ │ Cluster-C │ │ │ │ (Tenant A) │ │ (Tenant B) │ │ (Tenant C) │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │ │ │ Pros: Complete isolation, independent scaling │ │ Cons: Higher cost, operational overhead │ │ │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ Option 3: Quotas + ACLs on Shared Cluster │ │ ───────────────────────────────────────── │ │ │ │ ACLs for isolation + Quotas for fair sharing: │ │ │ │ kafka-configs.sh --alter --add-config \ │ │ 'producer_byte_rate=10485760,consumer_byte_rate=20971520' \ │ │ --entity-type users --entity-name tenant-a-app │ │ │ │ (Limits tenant-a to 10MB/s produce, 20MB/s consume) │ │ │ │ Pros: Efficient resource usage │ │ Cons: Noisy neighbor possible, complex ACL management │ │ │ └─────────────────────────────────────────────────────────────────────┘
5. Encryption at Rest
PROPERTIES(11 lines)CodeLoading syntax highlighter...
JAVA(31 lines)CodeLoading syntax highlighter...
Common Mistakes
Mistake 1: SASL/PLAIN Without TLS
PROPERTIES(8 lines)CodeLoading syntax highlighter...
Mistake 2: Overly Permissive ACLs
BASH(12 lines)CodeLoading syntax highlighter...
Mistake 3: Not Securing Inter-Broker Communication
PROPERTIES(7 lines)CodeLoading syntax highlighter...
Mistake 4: Hardcoding Credentials
JAVA(9 lines)CodeLoading syntax highlighter...
Mistake 5: Disabling Hostname Verification
JAVA(6 lines)CodeLoading syntax highlighter...
Debug This
Scenario: Authentication Failures
Symptoms:
SaslAuthenticationException: Authentication failed- Clients cannot connect
- No clear error message
Investigation:
BASH(19 lines)CodeLoading syntax highlighter...
JAVA(6 lines)CodeLoading syntax highlighter...
Common Causes:
- Wrong credentials: Typo in username/password
- Mechanism mismatch: Client uses PLAIN, broker expects SCRAM
- Missing JAAS config: JAAS configuration not provided
- SCRAM user not created: User exists in JAAS but not in ZK/KRaft
Resolution:
BASH(12 lines)CodeLoading syntax highlighter...
Exercises
Exercise 1: Set Up SASL/SCRAM Cluster
Configure a 3-broker cluster with:
- SASL/SCRAM-SHA-512 authentication
- TLS encryption
- At least 3 users with different permissions
- Verify authentication works
Exercise 2: Implement Role-Based ACLs
Design and implement ACLs for:
- Producer service (write to orders, read from inventory)
- Consumer service (read from orders, write to processed-orders)
- Admin (all operations)
- Read-only monitoring (describe only)
Exercise 3: Certificate Rotation
Implement certificate rotation:
- Generate new certificates
- Add new certs to truststore
- Update broker keystores
- Rolling restart with zero downtime
- Remove old certificates
Exercise 4: Security Audit Tool
Build a tool that:
- Lists all ACLs in the cluster
- Identifies overly permissive ACLs
- Finds users with no ACLs
- Generates compliance report
Exercise 5: Multi-Tenant Isolation
Set up multi-tenant environment:
- 3 tenants with prefix-based isolation
- Quotas per tenant
- Cross-tenant access denied
- Shared topics with ACLs
Interview Questions
Q1: Compare SASL/PLAIN, SASL/SCRAM, and mTLS for Kafka authentication.
A: Each has different strengths:
SASL/PLAIN:
- Username/password sent to broker
- Simple to set up
- MUST use with TLS (password in clear otherwise)
- Good for: Development, simple setups
- Drawback: Password stored in broker config
SASL/SCRAM (Salted Challenge Response):
- Password never sent over network
- Challenge-response protocol
- Credentials stored in ZK/KRaft (hashed)
- Good for: Production, dynamic user management
- Drawback: More complex setup
mTLS (Mutual TLS):
- Certificate-based authentication
- No passwords to manage
- Strong cryptographic identity
- Good for: Service-to-service, high security
- Drawback: Certificate management overhead, PKI required
Recommendation: SASL/SCRAM for most production uses, mTLS for service mesh environments.
Q2: How do Kafka ACLs work and what's the principle of least privilege?
A: ACLs control what authenticated users can do:
ACL Structure:
Principal + Permission + Operation + Resource + Pattern User:app1 + ALLOW + Write + topic:orders + LITERAL
Principle of least privilege:
- Grant minimum permissions needed
- Use specific topics, not wildcards
- Separate read and write permissions
- Use prefixed patterns sparingly
Example - Order service needs:
BASH(10 lines)CodeLoading syntax highlighter...
Q3: What's the difference between encryption in transit and at rest for Kafka?
A:
Encryption in transit (TLS):
- Protects data moving over network
- Client ↔ Broker, Broker ↔ Broker
- Configured via SSL listeners
- Prevents network sniffing
Encryption at rest:
- Protects data stored on disk
- Log segments, index files
- Usually OS/cloud level (LUKS, EBS encryption)
- Or application-level (encrypt before produce)
Both are needed for complete protection:
- Transit: Attacker can't intercept
- Rest: Attacker can't read stolen disks
Q4: How would you implement row-level security in Kafka?
A: Kafka doesn't have built-in row-level security. Approaches:
Option 1: Topic per tenant/permission level
orders-tenant-a orders-tenant-b
ACLs control access to topics.
Option 2: Application-level filtering
JAVA(7 lines)CodeLoading syntax highlighter...
Option 3: Kafka Streams filtering
JAVA(3 lines)CodeLoading syntax highlighter...
Option 4: Field-level encryption
JAVA(2 lines)CodeLoading syntax highlighter...
Q5: How do you secure a Kafka cluster in Kubernetes?
A: Multiple layers of security:
1. Network security:
YAML(12 lines)CodeLoading syntax highlighter...
2. Secrets management:
YAML(8 lines)CodeLoading syntax highlighter...
3. Service mesh (Istio/Linkerd):
- mTLS between all pods
- No Kafka-level TLS needed (but recommended)
4. Pod security:
YAML(3 lines)CodeLoading syntax highlighter...
5. Image security:
- Use trusted base images
- Scan for vulnerabilities
- Sign images
Summary
Key Takeaways
-
Three security layers: Encryption (TLS), Authentication (SASL/mTLS), Authorization (ACLs)
-
Always use TLS with SASL/PLAIN - passwords sent in clear otherwise
-
SASL/SCRAM recommended for production - dynamic user management, no clear passwords
-
Principle of least privilege - grant minimum ACLs needed for each service
-
Secure inter-broker communication, not just client connections
-
Never hardcode credentials - use environment variables or secrets managers
-
Multi-tenant isolation requires careful ACL design with prefixed patterns
-
Regular security audits - review ACLs, rotate credentials, update certificates
Quick Reference
Security Protocol Options
| Protocol | Encryption | Authentication |
|---|---|---|
| PLAINTEXT | No | No |
| SSL | Yes (TLS) | Optional (mTLS) |
| SASL_PLAINTEXT | No | Yes (SASL) |
| SASL_SSL | Yes (TLS) | Yes (SASL) |
Common ACL Commands
BASH(15 lines)CodeLoading syntax highlighter...
Spring Security Properties
YAML(11 lines)CodeLoading syntax highlighter...
Series Navigation
| Previous | Current | Next |
|---|---|---|
| Part 12: Schema Registry | Part 13: Security | Part 14: Monitoring |
Series Overview
- Part 0: How to Use This Series
- Parts 1-4: Fundamentals
- Parts 5-7: Producers
- Parts 8-11: Consumers
- Parts 12-14: Operations (Schema Registry, Security, Monitoring)
- Parts 15-17: Kafka Streams
- Parts 18-20: Patterns & Practices
- Part 21: Cheatsheet & Decision Guide