[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6},"actionData","errors","routes/blog-tag",{"_7":8,"_9":10},"tag",{"_33":89,"_74":90,"_37":91},"articles",{"_11":12,"_13":14,"_15":16,"_17":18,"_19":20,"_21":22,"_23":24,"_25":24,"_26":16,"_27":28},"content",[31,32],"pageable",{"_29":22,"_30":20},"totalElements",2,"totalPages",1,"size",10,"number",0,"first",true,"last","numberOfElements","empty",false,"page_number","page_size",{"_33":106,"_35":107,"_37":108,"_39":109,"_11":110,"_42":-5,"_43":24,"_44":28,"_45":46,"_47":111,"_49":112,"_51":52,"_53":113,"_55":113,"_56":57,"_58":59,"_60":114,"_62":113,"_63":115,"_64":116,"_66":117,"_68":118,"_70":119,"_72":120},{"_33":34,"_35":36,"_37":38,"_39":40,"_11":41,"_42":-5,"_43":24,"_44":28,"_45":46,"_47":48,"_49":50,"_51":52,"_53":54,"_55":54,"_56":57,"_58":59,"_60":61,"_62":54,"_63":57,"_64":65,"_66":67,"_68":69,"_70":71,"_72":73},"id",63,"title","Debugging Containers","slug","docker-part-13-debugging","excerpt","Debug crashed containers, nsenter, network debugging","## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| **Difficulty** | 🟡 Intermediate |\n| **Prerequisites** | Part 1 (Container Internals), Part 11 (Logging) |\n| **Key Tools** | exec, logs, inspect, nsenter, strace, tcpdump |\n| **Time Investment** | 30 minutes read + 60 minutes practice |\n| **Payoff** | Debug any container issue in minutes, not hours |\n\n---\n\n## 🎯 What You'll Learn\n\nAfter this article, you'll be able to:\n\n1. **Debug running containers** using exec, logs, and inspect effectively\n2. **Investigate crashed containers** and extract information from dead containers\n3. **Use advanced tools** like nsenter and strace for deep debugging\n4. **Debug network issues** with tcpdump and netstat inside containers\n5. **Perform post-mortem analysis** on containers that won't start\n\n---\n\n## 🔥 Production Story: The Heisenberg Container\n\n**The Setup**: A Java microservice crashes every few hours in production. When developers SSH into the server and exec into the container, it works fine. Only crashes when nobody's watching.\n\n**The Investigation**:\n```bash\n# Let's check recent crashes\ndocker ps -a | grep myapp\n# Container restarts: 47 in last 24 hours\n\n# Logs show nothing useful - JVM just disappears\ndocker logs myapp --tail 100\n# ... normal operation ...\n# (nothing, container just stops)\n\n# Check exit code\ndocker inspect myapp --format='{{.State.ExitCode}}'\n# 137 - killed by OOM!\n\n# But we set 2GB memory limit and JVM heap is 1GB...\ndocker inspect myapp --format='{{.HostConfig.Memory}}'\n# 2147483648 (2GB)\n\n# Check dmesg on host for OOM killer\ndmesg | grep -i \"killed process\" | tail -5\n# [timestamp] Killed process 12345 (java) total-vm:3145728kB\n# 3GB virtual memory? But we only set 1GB heap...\n```\n\n**The Root Cause**: JVM wasn't using container-aware settings. It was detecting 64GB host RAM and setting metaspace, code cache, and thread stacks accordingly. Total memory exceeded 2GB container limit.\n\n**The Fix**:\n```dockerfile\nENV JAVA_OPTS=\"-XX:+UseContainerSupport \\\n -XX:MaxRAMPercentage=75.0 \\\n -XX:+ExitOnOutOfMemoryError\"\n```\n\n**Lesson Learned**: Exit code 137 always means OOM. The container worked when developers exec'd because the JVM was already initialized - their debugging session didn't trigger more memory allocation.\n\n---\n\n## 🧠 Mental Model: The Debugging Hierarchy\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ DEBUGGING LEVELS │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ Level 1: OBSERVATION (90% of issues) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ docker logs → Application output │ │\n│ │ docker ps -a → Container state & exit codes │ │\n│ │ docker inspect → Full configuration │ │\n│ │ docker stats → Real-time resource usage │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ ↓ │\n│ Level 2: INTERACTION (8% of issues) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ docker exec → Run commands inside container │ │\n│ │ docker cp → Copy files in/out │ │\n│ │ docker diff → See filesystem changes │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ ↓ │\n│ Level 3: DEEP INSPECTION (2% of issues) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ nsenter → Enter container namespaces from host │ │\n│ │ strace → Trace system calls │ │\n│ │ tcpdump → Capture network traffic │ │\n│ │ /proc → Direct process inspection │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**Key Insight**: Always start at Level 1. Most issues are obvious from logs and exit codes. Only escalate when simpler tools don't reveal the problem.\n\n---\n\n## 🔬 Deep Dive\n\n### 1. Basic Debugging: logs, ps, inspect\n\n**Understanding Exit Codes**:\n\n```bash\n# Common exit codes and their meanings\n# 0 - Normal exit (application finished successfully)\n# 1 - Application error (generic failure)\n# 126 - Command not executable (permission issue)\n# 127 - Command not found (missing binary)\n# 137 - SIGKILL (OOM killer or docker kill)\n# 139 - SIGSEGV (segmentation fault)\n# 143 - SIGTERM (docker stop)\n\n# Check exit code\ndocker inspect myapp --format='{{.State.ExitCode}}'\n\n# Check if OOM killed\ndocker inspect myapp --format='{{.State.OOMKilled}}'\n\n# Full state information\ndocker inspect myapp --format='{{json .State}}' | jq\n```\n\n**Advanced Log Analysis**:\n\n```bash\n# Logs with timestamps\ndocker logs myapp --timestamps\n\n# Logs from specific time range\ndocker logs myapp --since \"2025-01-15T10:00:00\" --until \"2025-01-15T11:00:00\"\n\n# Follow logs with last 100 lines context\ndocker logs -f --tail 100 myapp\n\n# Logs from crashed container (important!)\ndocker logs $(docker ps -aq -f \"name=myapp\" -f \"status=exited\" | head -1)\n\n# JSON logs with jq\ndocker logs myapp 2>&1 | grep '^{' | jq 'select(.level == \"ERROR\")'\n```\n\n**Inspect Deep Dive**:\n\n```bash\n# Get specific fields with Go templates\ndocker inspect myapp --format='\nContainer: {{.Name}}\nStatus: {{.State.Status}}\nStarted: {{.State.StartedAt}}\nRestarts: {{.RestartCount}}\nOOM: {{.State.OOMKilled}}\nExit: {{.State.ExitCode}}\nError: {{.State.Error}}\n'\n\n# Network configuration\ndocker inspect myapp --format='{{json .NetworkSettings.Networks}}' | jq\n\n# Mounted volumes\ndocker inspect myapp --format='{{range .Mounts}}{{.Source}} -> {{.Destination}}\n{{end}}'\n\n# Environment variables\ndocker inspect myapp --format='{{range .Config.Env}}{{println .}}{{end}}'\n\n# Compare two containers\ndiff <(docker inspect container1) <(docker inspect container2)\n```\n\n---\n\n### 2. Interactive Debugging with exec\n\n**Basic exec Patterns**:\n\n```bash\n# Shell into container (prefer sh for minimal images)\ndocker exec -it myapp /bin/sh\ndocker exec -it myapp /bin/bash\n\n# Run as specific user\ndocker exec -u root -it myapp sh\ndocker exec -u www-data -it myapp sh\n\n# Run with environment variables\ndocker exec -e DEBUG=true -it myapp sh\n\n# Run in specific directory\ndocker exec -w /app -it myapp ls -la\n```\n\n**Debugging Without Shell** (distroless images):\n\n```bash\n# Distroless has no shell - use debug image\n# Option 1: Sidecar debug container sharing PID namespace\ndocker run -it --pid=container:myapp --net=container:myapp \\\n --cap-add SYS_PTRACE \\\n nicolaka/netshoot\n\n# Option 2: Copy binary from another container\ndocker cp debug-container:/usr/bin/strace - | docker cp - myapp:/tmp/\n\n# Option 3: Use docker debug (Docker Desktop feature)\ndocker debug myapp\n```\n\n**Process Inspection**:\n\n```bash\n# List processes\ndocker exec myapp ps aux\n\n# Check file descriptors (useful for connection leaks)\ndocker exec myapp ls -la /proc/1/fd | wc -l\n\n# Check memory maps\ndocker exec myapp cat /proc/1/maps | head -20\n\n# Check environment of PID 1\ndocker exec myapp cat /proc/1/environ | tr '\\0' '\\n'\n\n# Check open files\ndocker exec myapp cat /proc/1/fd/ 2>/dev/null | head\n```\n\n---\n\n### 3. Network Debugging\n\n**Container Network Analysis**:\n\n```bash\n# Check network interfaces\ndocker exec myapp ip addr\ndocker exec myapp cat /etc/resolv.conf\n\n# Test DNS resolution\ndocker exec myapp nslookup api-service\ndocker exec myapp getent hosts api-service\n\n# Test connectivity\ndocker exec myapp ping -c 3 api-service\ndocker exec myapp curl -v http://api-service:8080/health\n\n# Check listening ports\ndocker exec myapp netstat -tlnp\ndocker exec myapp ss -tlnp\n\n# Check established connections\ndocker exec myapp netstat -anp | grep ESTABLISHED\n```\n\n**Using netshoot for Deep Network Debug**:\n\n```bash\n# netshoot is the Swiss Army knife for network debugging\ndocker run -it --net=container:myapp nicolaka/netshoot\n\n# Inside netshoot:\n# DNS debugging\ndig api-service\ndrill api-service\n\n# TCP connection debugging\nnc -zv api-service 8080\n\n# HTTP debugging\ncurl -v --trace-time http://api-service:8080/\n\n# Packet capture\ntcpdump -i eth0 port 8080 -w /tmp/capture.pcap\n\n# Connection tracking\nconntrack -L\n\n# Network path\nmtr api-service\n```\n\n**Debugging DNS Issues**:\n\n```bash\n# Common DNS problems and solutions\n\n# Check if DNS server responds\ndocker exec myapp nslookup google.com 127.0.0.11\n\n# Check DNS configuration\ndocker exec myapp cat /etc/resolv.conf\n\n# Test with external DNS\ndocker exec myapp nslookup api-service 8.8.8.8\n\n# Debug DNS with verbose output\ndocker run --net=container:myapp nicolaka/netshoot \\\n dig +trace api-service\n\n# Check if embedded DNS is working\ndocker exec myapp getent ahosts api-service\n```\n\n---\n\n### 4. Advanced Debugging: nsenter\n\n**nsenter** lets you enter container namespaces directly from the host - essential when container has no debugging tools.\n\n```bash\n# Get container's main process PID\nCONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' myapp)\n\n# Enter all namespaces of container\nsudo nsenter -t $CONTAINER_PID -a\n\n# Enter specific namespaces\nsudo nsenter -t $CONTAINER_PID -n ip addr # Network namespace\nsudo nsenter -t $CONTAINER_PID -m ls /app # Mount namespace\nsudo nsenter -t $CONTAINER_PID -p ps aux # PID namespace\n\n# Run host tools in container's network namespace\nsudo nsenter -t $CONTAINER_PID -n tcpdump -i eth0\n\n# Run host tools in container's filesystem\nsudo nsenter -t $CONTAINER_PID -m cat /app/config.yml\n```\n\n**Namespace-specific Debugging**:\n\n```bash\n# Get PID for namespace operations\nCONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' myapp)\n\n# Compare container and host namespaces\nls -la /proc/$CONTAINER_PID/ns/\nls -la /proc/1/ns/\n\n# Network debugging without exec\nsudo nsenter -t $CONTAINER_PID -n ss -tlnp\nsudo nsenter -t $CONTAINER_PID -n ip route\nsudo nsenter -t $CONTAINER_PID -n iptables -L -n\n\n# Filesystem debugging\nsudo nsenter -t $CONTAINER_PID -m df -h\nsudo nsenter -t $CONTAINER_PID -m find /app -name \"*.log\" -mmin -5\n```\n\n---\n\n### 5. System Call Tracing with strace\n\n**Basic strace Usage**:\n\n```bash\n# Trace a process inside container\ndocker exec myapp strace -p 1 -f\n\n# If container lacks strace, use nsenter\nCONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' myapp)\nsudo strace -p $CONTAINER_PID -f\n\n# Trace specific system calls\nsudo strace -p $CONTAINER_PID -e trace=open,read,write\n\n# Trace network calls\nsudo strace -p $CONTAINER_PID -e trace=network\n\n# Trace file operations\nsudo strace -p $CONTAINER_PID -e trace=file\n\n# Save output to file\nsudo strace -p $CONTAINER_PID -o /tmp/strace.log -f\n```\n\n**Common strace Patterns**:\n\n```bash\n# Find what files an app is looking for\nsudo strace -p $CONTAINER_PID -e trace=open,openat 2>&1 | grep ENOENT\n\n# Find what's blocking the process\nsudo strace -p $CONTAINER_PID -e trace=futex,epoll_wait\n\n# Check socket operations\nsudo strace -p $CONTAINER_PID -e trace=socket,connect,bind,listen,accept\n\n# Trace with timing information\nsudo strace -p $CONTAINER_PID -T -e trace=read,write\n\n# Trace and filter specific path\nsudo strace -p $CONTAINER_PID 2>&1 | grep \"/etc/config\"\n```\n\n**Debugging Startup Issues**:\n\n```bash\n# Run container with strace from start\ndocker run --cap-add SYS_PTRACE --security-opt seccomp=unconfined \\\n myimage strace -f /entrypoint.sh\n\n# Or trace the entrypoint\ndocker run --entrypoint strace myimage -f /original-entrypoint.sh\n```\n\n---\n\n### 6. Debugging Crashed Containers\n\n**Extract Information from Dead Containers**:\n\n```bash\n# Container that exited still exists\ndocker ps -a | grep myapp\n\n# Get logs from dead container\ndocker logs myapp\n\n# Copy files out of dead container\ndocker cp myapp:/var/log/app.log ./\ndocker cp myapp:/app/config.yml ./\n\n# See filesystem changes (what the app modified)\ndocker diff myapp\n\n# Create image from dead container for analysis\ndocker commit myapp myapp-debug\ndocker run -it myapp-debug /bin/sh\n\n# Export entire filesystem\ndocker export myapp > myapp-filesystem.tar\ntar -tf myapp-filesystem.tar | grep -E \"\\.(log|conf|yml)$\"\n```\n\n**Debugging Containers That Won't Start**:\n\n```bash\n# Override entrypoint to get shell\ndocker run -it --entrypoint /bin/sh myimage\n\n# Override command\ndocker run -it myimage /bin/sh\n\n# Check if entrypoint exists and is executable\ndocker run --entrypoint ls myimage -la /entrypoint.sh\n\n# Run with verbose output\ndocker run -e DEBUG=1 -e TRACE=1 myimage\n\n# Check for missing libraries\ndocker run --entrypoint ldd myimage /app/binary\n\n# Run strace on startup\ndocker run --cap-add SYS_PTRACE --security-opt seccomp=unconfined \\\n --entrypoint strace myimage -f /entrypoint.sh\n```\n\n---\n\n### 7. Resource Debugging\n\n**Memory Issues**:\n\n```bash\n# Real-time stats\ndocker stats myapp --no-stream\n\n# Memory details from cgroup\ndocker exec myapp cat /sys/fs/cgroup/memory.current\ndocker exec myapp cat /sys/fs/cgroup/memory.max\n\n# Check OOM events\ndocker exec myapp cat /sys/fs/cgroup/memory.events\n\n# Host-level OOM check\ndmesg | grep -i \"killed process\"\njournalctl -k | grep -i oom\n\n# Memory breakdown for JVM\ndocker exec myapp jcmd 1 VM.native_memory summary\n```\n\n**CPU Issues**:\n\n```bash\n# CPU stats\ndocker stats myapp --format \"{{.CPUPerc}}\"\n\n# Check CPU throttling\ndocker exec myapp cat /sys/fs/cgroup/cpu.stat\n\n# Throttled periods indicate CPU limit issues\n# nr_throttled: X (should be 0 or low)\n\n# Check process CPU usage\ndocker exec myapp top -b -n 1\n\n# Profile CPU with perf (requires capability)\ndocker run --cap-add SYS_ADMIN --security-opt seccomp=unconfined \\\n -v /path/to/perf:/perf myimage perf top\n```\n\n**Disk I/O Issues**:\n\n```bash\n# Check disk usage\ndocker exec myapp df -h\n\n# Check I/O stats\ndocker exec myapp cat /sys/fs/cgroup/io.stat\n\n# Find large files\ndocker exec myapp find / -type f -size +100M 2>/dev/null\n\n# Check what's writing\ndocker exec myapp iotop -b -n 1\n```\n\n---\n\n### 8. Complete Debug Script\n\n```bash\n#!/bin/bash\n# container-debug.sh - Comprehensive container debugging\n\nCONTAINER=$1\n\nif [ -z \"$CONTAINER\" ]; then\n echo \"Usage: $0 \"\n exit 1\nfi\n\necho \"═══════════════════════════════════════════════════════════\"\necho \"Container Debug Report: $CONTAINER\"\necho \"Generated: $(date)\"\necho \"═══════════════════════════════════════════════════════════\"\n\n# Basic info\necho -e \"\\n### Container State ###\"\ndocker inspect $CONTAINER --format='\nID: {{.Id}}\nName: {{.Name}}\nImage: {{.Config.Image}}\nStatus: {{.State.Status}}\nRunning: {{.State.Running}}\nPid: {{.State.Pid}}\nStarted: {{.State.StartedAt}}\nFinished: {{.State.FinishedAt}}\nExit Code: {{.State.ExitCode}}\nOOM Killed: {{.State.OOMKilled}}\nRestarts: {{.RestartCount}}\nError: {{.State.Error}}\n'\n\n# Exit code interpretation\nEXIT_CODE=$(docker inspect $CONTAINER --format='{{.State.ExitCode}}')\necho -e \"\\n### Exit Code Analysis ###\"\ncase $EXIT_CODE in\n 0) echo \"Exit 0: Normal termination\" ;;\n 1) echo \"Exit 1: Application error\" ;;\n 126) echo \"Exit 126: Command not executable (permission issue)\" ;;\n 127) echo \"Exit 127: Command not found\" ;;\n 137) echo \"Exit 137: SIGKILL - OOM killer or forced stop\" ;;\n 139) echo \"Exit 139: SIGSEGV - Segmentation fault\" ;;\n 143) echo \"Exit 143: SIGTERM - Graceful shutdown\" ;;\n *) echo \"Exit $EXIT_CODE: Check application logs\" ;;\nesac\n\n# Resource usage (if running)\nif docker inspect $CONTAINER --format='{{.State.Running}}' | grep -q true; then\n echo -e \"\\n### Resource Usage ###\"\n docker stats $CONTAINER --no-stream --format \\\n \"CPU: {{.CPUPerc}}\\nMemory: {{.MemUsage}}\\nNet: {{.NetIO}}\\nDisk: {{.BlockIO}}\"\n\n echo -e \"\\n### Process List ###\"\n docker exec $CONTAINER ps aux 2>/dev/null || echo \"(Cannot exec into container)\"\n\n echo -e \"\\n### Network ###\"\n docker exec $CONTAINER cat /etc/resolv.conf 2>/dev/null\n docker exec $CONTAINER ss -tlnp 2>/dev/null || docker exec $CONTAINER netstat -tlnp 2>/dev/null\nfi\n\n# Logs\necho -e \"\\n### Recent Logs (last 50 lines) ###\"\ndocker logs $CONTAINER --tail 50 --timestamps 2>&1\n\n# Mounts\necho -e \"\\n### Mounted Volumes ###\"\ndocker inspect $CONTAINER --format='{{range .Mounts}}{{.Type}}: {{.Source}} -> {{.Destination}} ({{.Mode}})\n{{end}}'\n\n# Network\necho -e \"\\n### Network Configuration ###\"\ndocker inspect $CONTAINER --format='{{range $net, $config := .NetworkSettings.Networks}}\nNetwork: {{$net}}\n IP: {{$config.IPAddress}}\n Gateway: {{$config.Gateway}}\n MAC: {{$config.MacAddress}}\n{{end}}'\n\n# Environment (redact sensitive)\necho -e \"\\n### Environment Variables (sensitive redacted) ###\"\ndocker inspect $CONTAINER --format='{{range .Config.Env}}{{println .}}{{end}}' | \\\n sed -E 's/(PASSWORD|SECRET|KEY|TOKEN)=.*/\\1=[REDACTED]/gi'\n\n# Health check\necho -e \"\\n### Health Check Status ###\"\ndocker inspect $CONTAINER --format='\nConfigured: {{if .Config.Healthcheck}}Yes{{else}}No{{end}}\nStatus: {{.State.Health.Status}}\nFailing Streak: {{.State.Health.FailingStreak}}\n' 2>/dev/null || echo \"No health check configured\"\n\n# Filesystem changes\necho -e \"\\n### Filesystem Changes ###\"\ndocker diff $CONTAINER 2>/dev/null | head -20 || echo \"(Container not available)\"\n\necho -e \"\\n═══════════════════════════════════════════════════════════\"\necho \"Debug report complete\"\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Ignoring Exit Codes\n\n```bash\n# ❌ Wrong: Just restarting without investigating\ndocker restart myapp\n\n# ✅ Right: Check exit code first\nEXIT=$(docker inspect myapp --format='{{.State.ExitCode}}')\nOOM=$(docker inspect myapp --format='{{.State.OOMKilled}}')\necho \"Exit: $EXIT, OOM: $OOM\"\ndocker logs myapp --tail 100\n# Then investigate based on findings\n```\n\n### Mistake 2: Debugging in Production Without Caution\n\n```bash\n# ❌ Wrong: Running strace in production\nsudo strace -p $PROD_PID -f # Can slow down the process!\n\n# ✅ Right: Use sampling or targeted tracing\n# Only trace specific syscalls for short duration\nsudo timeout 5 strace -p $PROD_PID -e trace=open,connect -c\n\n# Or use debug container that doesn't affect main process\ndocker run --net=container:prod-app nicolaka/netshoot tcpdump -c 100\n```\n\n### Mistake 3: Not Preserving Dead Container State\n\n```bash\n# ❌ Wrong: Immediately removing failed container\ndocker rm myapp # Lost all debugging info!\n\n# ✅ Right: Extract info first\ndocker logs myapp > /tmp/myapp-crash.log\ndocker cp myapp:/app/logs/ /tmp/myapp-logs/\ndocker diff myapp > /tmp/myapp-changes.txt\ndocker commit myapp myapp-postmortem # Optional: save state\ndocker rm myapp # Now safe to remove\n```\n\n### Mistake 4: Exec-ing as Wrong User\n\n```bash\n# ❌ Wrong: Always using root\ndocker exec -u root -it myapp sh\n# Might not see the same permission issues app has\n\n# ✅ Right: Match the app's user\ndocker exec -it myapp sh # Uses container's default user\ndocker exec -it myapp whoami # Check which user that is\n# Only use root when specifically needed\n```\n\n### Mistake 5: Missing Container Restart Patterns\n\n```bash\n# ❌ Wrong: Not checking restart history\ndocker ps # Shows current status only\n\n# ✅ Right: Check restart count and pattern\ndocker inspect myapp --format='\nRestarts: {{.RestartCount}}\nLast Started: {{.State.StartedAt}}\n'\n\n# Check events for restart pattern\ndocker events --filter container=myapp --since \"1h\" | grep -E \"(die|start)\"\n```\n\n---\n\n## 🐛 Debug This\n\nYou're on-call and get paged. Container keeps restarting:\n\n```bash\n$ docker ps\nCONTAINER ID STATUS NAMES\nabc123 Restarting (137) 2 sec ago api-server\n\n$ docker logs api-server --tail 20\nStarting application...\nLoading configuration...\nConnecting to database...\nConnection established\nStarting HTTP server on port 8080\nServer ready to accept connections\n# (no error message, logs just stop)\n\n$ docker inspect api-server --format='{{.State.OOMKilled}}'\nfalse\n\n$ docker stats api-server --no-stream\nNAME CPU % MEM USAGE / LIMIT\napi-server 0.50% 1.2GiB / 2GiB\n```\n\nExit code 137 but OOMKilled is false. Memory at 60% of limit. What's happening?\n\n

Click to reveal investigation

\n\n**Analysis**:\n1. Exit 137 = SIGKILL, but not from OOM killer\n2. Memory usage (1.2GB) well under limit (2GB)\n3. Logs show normal startup, no error\n\n**Possible causes of non-OOM SIGKILL**:\n- Health check failing → Docker kills container\n- `docker kill` or `docker stop` timeout\n- Kubernetes liveness probe failing\n- External process killing the container\n\n**Investigation**:\n```bash\n# Check if health check is configured\ndocker inspect api-server --format='{{json .Config.Healthcheck}}' | jq\n# Result: Health check hitting /health endpoint\n\n# Check health check status\ndocker inspect api-server --format='{{json .State.Health}}' | jq\n# Result: Status: \"unhealthy\", FailingStreak: 3\n\n# Check what health check returns\ndocker exec api-server curl -v localhost:8080/health\n# Result: Connection refused\n\n# The server says \"ready\" but isn't actually listening!\n# Check if port is bound\ndocker exec api-server netstat -tlnp | grep 8080\n# Nothing on 8080!\n```\n\n**Root Cause**: The application logs \"Server ready\" before the socket is actually bound. Health check starts immediately, can't connect, marks unhealthy, Docker kills container.\n\n**Fix**: Either fix the application to not log \"ready\" until socket is bound, or add startup grace period:\n```yaml\nhealthcheck:\n test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8080/health\"]\n start_period: 30s # Don't fail during first 30 seconds\n interval: 10s\n timeout: 5s\n retries: 3\n```\n

\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Basic Debug Investigation\nDeploy a container with an intentional issue and debug it:\n```bash\ndocker run -d --name debug-ex1 nginx:alpine\ndocker exec debug-ex1 rm /etc/nginx/nginx.conf\ndocker restart debug-ex1\n# Debug: Why won't it start? Use logs, inspect, and diff\n```\n\n### Exercise 2: Network Debugging\nCreate two containers that should communicate but can't:\n```bash\ndocker network create debug-net\ndocker run -d --name server --network debug-net nginx\ndocker run -d --name client --network bridge alpine sleep 3600\n# Debug: Why can't client reach server?\n# Use exec, inspect networks, and netshoot\n```\n\n### Exercise 3: Memory Investigation\n```bash\ndocker run -d --name memtest --memory=100m python:3.11 python -c \"\nimport time\ndata = []\nwhile True:\n data.append('x' * 1024 * 1024) # 1MB per second\n time.sleep(1)\n\"\n# Monitor with docker stats, predict when it will be killed\n# Investigate with inspect after it dies\n```\n\n### Exercise 4: Deep Debug with nsenter\n```bash\ndocker run -d --name minimal gcr.io/distroless/static-debian12 sleep 3600 || \\\ndocker run -d --name minimal busybox sleep 3600\n# Use nsenter from host to:\n# - List processes in container\n# - Check network configuration\n# - Explore filesystem\n# (Without exec, which won't work on distroless)\n```\n\n### Exercise 5: Post-mortem Analysis\n```bash\ndocker run --name postmortem alpine sh -c \"\n echo 'starting' > /tmp/startup.log\n sleep 2\n echo 'doing work' >> /tmp/startup.log\n exit 42\n\"\n# After it exits:\n# - Extract logs and files\n# - Determine what happened\n# - Create debug image for investigation\n```\n\n---\n\n## 🎤 Interview Questions\n\n### Q1: A container exits with code 137. What does this mean and how do you investigate?\n\n**Answer**: Exit code 137 indicates the process received SIGKILL (128 + 9). This typically means:\n\n1. **OOM Killer** - Most common cause:\n```bash\ndocker inspect container --format='{{.State.OOMKilled}}'\ndmesg | grep -i \"killed process\"\n```\n\n2. **Health check failure** - Docker kills unhealthy containers:\n```bash\ndocker inspect container --format='{{json .State.Health}}' | jq\ndocker inspect container --format='{{.Config.Healthcheck}}'\n```\n\n3. **External kill** - `docker kill`, orchestrator, or system process:\n```bash\ndocker events --filter container=name --since \"1h\"\njournalctl -u docker | grep container-name\n```\n\n4. **Timeout during stop** - Container didn't respond to SIGTERM:\n```bash\ndocker inspect container --format='{{.Config.StopTimeout}}'\n```\n\nInvestigation steps:\n1. Check OOMKilled flag first\n2. Check health status if configured\n3. Check docker events for who killed it\n4. Check dmesg and journalctl for system-level kills\n\n---\n\n### Q2: How do you debug a container that uses a distroless base image?\n\n**Answer**: Distroless images have no shell or debugging tools. Strategies:\n\n1. **Sidecar debug container** (production-safe):\n```bash\n# Share PID and network namespace with debug container\ndocker run -it --pid=container:target --net=container:target \\\n --cap-add SYS_PTRACE nicolaka/netshoot\n```\n\n2. **nsenter from host** (requires host access):\n```bash\nCONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' target)\nsudo nsenter -t $CONTAINER_PID -n ip addr # Network\nsudo nsenter -t $CONTAINER_PID -m ls /app # Filesystem\n```\n\n3. **Debug variant images** (for development):\n```dockerfile\n# Some distroless have debug variants\nFROM gcr.io/distroless/java17-debian12:debug\n```\n\n4. **Build debug layer** (for your images):\n```dockerfile\nFROM base as release\n# ... production image\n\nFROM release as debug\nCOPY --from=busybox /bin/sh /bin/sh\nCOPY --from=busybox /bin/busybox /bin/busybox\n```\n\n5. **Export and analyze filesystem**:\n```bash\ndocker export container > fs.tar\ntar -tf fs.tar | grep -E \"(log|conf|err)\"\n```\n\n---\n\n### Q3: What's the difference between docker logs and checking /var/log inside the container?\n\n**Answer**: They serve different purposes:\n\n**docker logs**:\n- Captures stdout/stderr from PID 1 (and children if properly configured)\n- Managed by Docker logging driver\n- Persists based on log rotation settings\n- Works even after container stops\n- Example: `docker logs --since 5m container`\n\n**/var/log inside container**:\n- Application-specific log files\n- Not captured by Docker logging\n- Lost when container is removed (unless volume mounted)\n- Requires exec or cp to access\n- Example: `docker exec container cat /var/log/app/error.log`\n\n**Best practices**:\n1. **12-factor apps** should log to stdout/stderr for Docker to capture\n2. **Legacy apps** writing to files need log forwarders or volume mounts\n3. **Production** should use centralized logging (ELK, Loki, CloudWatch)\n\n```bash\n# Check both sources\ndocker logs container --tail 50 # stdout/stderr\ndocker exec container ls -la /var/log/ # File-based logs\ndocker cp container:/var/log/app/ ./logs/ # Extract file logs\n```\n\n---\n\n### Q4: How would you trace why a containerized application can't connect to an external service?\n\n**Answer**: Systematic network debugging approach:\n\n```bash\n# 1. Verify DNS resolution\ndocker exec container nslookup external-service.com\ndocker exec container getent hosts external-service.com\n\n# 2. Test basic connectivity\ndocker exec container ping -c 3 external-service.com\ndocker exec container nc -zv external-service.com 443\n\n# 3. Check route and firewall\ndocker exec container ip route\ndocker exec container traceroute external-service.com\n\n# 4. Test actual HTTP/HTTPS\ndocker exec container curl -v https://external-service.com/\n\n# 5. If tools missing, use netshoot sidecar\ndocker run -it --net=container:myapp nicolaka/netshoot\n\n# 6. Packet capture for deep analysis\ndocker run --net=container:myapp nicolaka/netshoot \\\n tcpdump -i eth0 host external-service.com -w /tmp/capture.pcap\n\n# 7. Check from host perspective\nCONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' myapp)\nsudo nsenter -t $CONTAINER_PID -n curl -v https://external-service.com/\n```\n\nCommon causes:\n- DNS resolution failure (check /etc/resolv.conf)\n- Network policy/firewall blocking egress\n- Proxy configuration missing\n- TLS/certificate issues\n- Container network mode wrong (host vs bridge)\n\n---\n\n### Q5: Explain how you would perform post-mortem analysis on a container that crashed and won't restart.\n\n**Answer**: Systematic post-mortem process:\n\n```bash\n# 1. Preserve the evidence before anyone removes it\ndocker logs crashed-container > /tmp/crash-logs.txt 2>&1\ndocker inspect crashed-container > /tmp/crash-inspect.json\ndocker diff crashed-container > /tmp/crash-diff.txt\n\n# 2. Analyze exit information\nEXIT_CODE=$(docker inspect crashed-container --format='{{.State.ExitCode}}')\nOOM=$(docker inspect crashed-container --format='{{.State.OOMKilled}}')\nERROR=$(docker inspect crashed-container --format='{{.State.Error}}')\necho \"Exit: $EXIT_CODE, OOM: $OOM, Error: $ERROR\"\n\n# 3. Check host-level events\ndmesg | tail -50\njournalctl -u docker --since \"1 hour ago\" | grep crashed-container\n\n# 4. Extract application data\ndocker cp crashed-container:/app/logs/ ./crash-analysis/\ndocker cp crashed-container:/tmp/ ./crash-analysis/tmp/\n\n# 5. Create debug image for investigation\ndocker commit crashed-container crashed-container-debug\ndocker run -it --entrypoint /bin/sh crashed-container-debug\n\n# 6. If container won't start, override entrypoint\ndocker run -it --entrypoint /bin/sh original-image\n# Then manually trace startup:\nstrace -f /original/entrypoint.sh\n\n# 7. Export full filesystem for offline analysis\ndocker export crashed-container > crashed-fs.tar\ntar -xf crashed-fs.tar -C ./crash-analysis/filesystem/\n```\n\nKey things to look for:\n- Exit code meaning (137=SIGKILL, 139=SIGSEGV, etc.)\n- Last log entries before crash\n- Modified files (docker diff)\n- Core dumps if segfault\n- Resource exhaustion evidence\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Debugging Hierarchy\n1. **Level 1**: logs, ps, inspect, stats - solves 90% of issues\n2. **Level 2**: exec, cp, diff - interactive investigation\n3. **Level 3**: nsenter, strace, tcpdump - deep system analysis\n\n### Exit Code Quick Reference\n| Code | Signal | Meaning |\n|------|--------|---------|\n| 0 | - | Normal exit |\n| 1 | - | Application error |\n| 137 | SIGKILL | OOM or forced kill |\n| 139 | SIGSEGV | Segmentation fault |\n| 143 | SIGTERM | Graceful shutdown |\n\n### Essential Commands\n\n```bash\n# Investigation\ndocker logs container --tail 100 --timestamps\ndocker inspect container --format='{{json .State}}'\ndocker stats container --no-stream\n\n# Interaction\ndocker exec -it container sh\ndocker cp container:/path ./local/\n\n# Deep debug\nnsenter -t $PID -n ip addr\nstrace -p $PID -e trace=network\n\n# Post-mortem\ndocker diff container\ndocker commit container debug-image\ndocker export container > filesystem.tar\n```\n\n---\n\n## 📋 Quick Reference\n\n```bash\n# Exit code check\ndocker inspect $C --format='Exit:{{.State.ExitCode}} OOM:{{.State.OOMKilled}}'\n\n# Logs with timestamp\ndocker logs $C --timestamps --since \"10m\"\n\n# Process in container\ndocker exec $C ps aux\n\n# Network debug\ndocker run -it --net=container:$C nicolaka/netshoot\n\n# Namespace entry\nPID=$(docker inspect --format '{{.State.Pid}}' $C)\nsudo nsenter -t $PID -a\n\n# Extract from dead container\ndocker cp $C:/path ./local/\ndocker commit $C $C-debug\n\n# Trace syscalls\nsudo strace -p $PID -f -e trace=file\n```\n\n---\n\n## 📅 Review Schedule\n\nMaster debugging through spaced repetition:\n- **Day 1**: Practice logs, ps, inspect on running containers\n- **Day 3**: Debug intentionally broken containers\n- **Day 7**: Network debugging exercises\n- **Day 14**: nsenter and strace deep dive\n- **Day 30**: Full post-mortem analysis on complex failure\n\n---\n\n## 📚 Series Navigation\n\n- **Previous**: [Part 12 - Container Security Hardening](/articles/docker-part-12-security)\n- **Next**: [Part 14 - Compose File Deep Dive](/articles/docker-part-14-compose-deep-dive)\n- **Index**: [Docker Compendium Series](/series/docker-compendium)","featuredImage","isPublished","isFeatured","subscriptionType","BASIC","publishedAt","2026-06-10T07:19:25.513670Z","createdAt","2026-06-10T07:19:25.514302Z","updatedAt","2026-06-16T21:26:56.086200Z","readingTimeMinutes",19,"estimatedReadingTime","authorId",3,"authorUsername","tomasz_main","categoryName","Devops","readingTime","seriesId","seriesOrder",13,"author",{"_33":57,"_96":59,"_97":98,"_99":100,"_101":-5,"_102":-5,"_103":-5,"_104":-5,"_105":-5},"category",{"_33":92,"_74":61,"_37":93,"_77":94,"_95":-5},"tags",[80,81,82],"series",{"_33":57,"_74":75,"_37":76,"_77":78,"_79":24},"name","Docker Compendium: From Practitioner to Expert","docker-compendium","description","Master Docker internals, optimization patterns, and production best practices. For developers who use Docker daily but want deeper understanding of what's happening under the hood. Not a beginner series - we assume you can write Dockerfiles and use docker-compose. Learn what namespaces and cgroups really do, why your builds are slow, how to debug containers that crash, and how to run Docker in production like a pro.\n","isActive",{"_33":89,"_74":90,"_37":91},{"_33":86,"_74":87,"_37":88},{"_33":83,"_74":84,"_37":85},241,"Troubleshooting","troubleshooting",213,"Docker","docker",240,"Debugging","debugging",7,"devops","Category for Devops","colorHex","username","firstName","Tomasz","lastName","Jarosz","profilePicture","bio","website","githubUrl","linkedinUrl",123,"Testing and Debugging Collections","java-collections-part-27-testing-debugging","Build comprehensive collection test suites with AssertJ/Hamcrest, debug ConcurrentModificationException, diagnose memory leaks, and test thread-safety.","# Java Collections Compendium Part 27: Testing & Debugging Collections\n\nMaster collection testing with AssertJ and Hamcrest, debug ConcurrentModificationException and memory leaks, and build comprehensive test suites for collection-heavy code.\n\n## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| **Topic** | AssertJ assertions, Hamcrest matchers, debugging techniques, memory analysis |\n| **Complexity** | Intermediate |\n| **Prerequisites** | Part 1 (Collection Architecture), JUnit basics |\n| **Time to Master** | 3-4 hours |\n| **Interview Frequency** | Medium (testing knowledge valued) |\n\n## 🎯 What You'll Learn\n\nAfter completing this article, you will be able to:\n\n1. Write expressive collection tests with AssertJ\n2. Debug ConcurrentModificationException effectively\n3. Find and fix collection-related memory leaks\n4. Test thread-safety of concurrent collections\n5. Use IntelliJ debugger features for collection inspection\n\n---\n\n## Production Story: The Test That Passed But Shouldn't Have\n\n### The Incident\n\nOur test suite was green, but production had duplicates in what should be a unique collection:\n\n```java\n// ProductService.java\npublic Set getActiveSkus() {\n return products.stream()\n .filter(Product::isActive)\n .map(Product::getSku)\n .collect(Collectors.toSet());\n}\n\n// ProductServiceTest.java - The misleading test\n@Test\nvoid shouldReturnActiveSkus() {\n // Setup: Products with SKUs\n productService.addProduct(new Product(\"SKU-001\", true));\n productService.addProduct(new Product(\"SKU-002\", true));\n productService.addProduct(new Product(\"SKU-003\", false));\n\n Set activeSkus = productService.getActiveSkus();\n\n // ❌ This test passes even with duplicates!\n assertEquals(2, activeSkus.size());\n assertTrue(activeSkus.contains(\"SKU-001\"));\n assertTrue(activeSkus.contains(\"SKU-002\"));\n}\n```\n\n### The Problem\n\n```text\nThe test passed, but production failed because:\n\nActual production data:\n├─ Product(SKU-001, active=true)\n├─ Product(SKU-001, active=true) ← Duplicate SKU!\n├─ Product(SKU-002, active=true)\n└─ Product(SKU-003, active=false)\n\nExpected behavior: Error or deduplicate\nActual behavior: Silent deduplication (Set semantics)\n\nThe test didn't catch this because:\n├─ Test data had no duplicates\n├─ Assertions only checked contains, not exact content\n└─ No assertion verified \"no unexpected elements\"\n\nProduction Impact:\n├─ Inventory count wrong (2 vs 3 active products)\n├─ Orders shipped to wrong quantities\n├─ 50 customer complaints\n└─ Manual inventory reconciliation needed\n```\n\n### The Better Test\n\n```java\n// ProductServiceTest.java - Comprehensive test\n@Test\nvoid shouldReturnExactlyActiveSkus() {\n // Setup with realistic data (including potential duplicates)\n productService.addProduct(new Product(\"SKU-001\", true));\n productService.addProduct(new Product(\"SKU-001\", true)); // Duplicate!\n productService.addProduct(new Product(\"SKU-002\", true));\n productService.addProduct(new Product(\"SKU-003\", false));\n\n Set activeSkus = productService.getActiveSkus();\n\n // ✅ AssertJ: Exact content verification\n assertThat(activeSkus)\n .containsExactlyInAnyOrder(\"SKU-001\", \"SKU-002\")\n .hasSize(2)\n .doesNotContain(\"SKU-003\");\n\n // ✅ If duplicates should cause error:\n assertThatThrownBy(() -> productService.getActiveSkusStrict())\n .isInstanceOf(DuplicateSkuException.class)\n .hasMessageContaining(\"SKU-001\");\n}\n\n@Test\nvoid shouldDetectDuplicateSkus() {\n productService.addProduct(new Product(\"SKU-001\", true));\n productService.addProduct(new Product(\"SKU-001\", true));\n\n // Verify duplicate detection\n List duplicates = productService.findDuplicateSkus();\n assertThat(duplicates).containsExactly(\"SKU-001\");\n}\n```\n\n---\n\n## Mental Model: Test Coverage Pyramid for Collections\n\n```text\n┌─────────────────────────────────────────────────────────────────────┐\n│ COLLECTION TEST COVERAGE │\n├─────────────────────────────────────────────────────────────────────┤\n│ │\n│ ┌───────────┐ │\n│ │ E2E/Perf │ Few: Full system behavior │\n│ /│ │\\ - Memory under load │\n│ / └───────────┘ \\ - Concurrent access │\n│ / Integration \\ │\n│ /┌─────────────────┐\\ │\n│ / │ Service + Repo │ \\ Some: Component behavior │\n│ / │ │ \\ - Repository queries │\n│ / └─────────────────┘ \\ - Service layer logic │\n│ / Unit Tests \\ │\n│ /┌───────────────────────────┐\\ │\n│ / │ Individual methods │ \\ Many: Method behavior│\n│ / │ - Edge cases (empty, null) │ \\ │\n│ / │ - Boundary conditions │ \\ │\n│ / │ - All collection operations│ \\ │\n│ / └───────────────────────────┘ \\ │\n│ │\n│ TEST DIMENSIONS FOR COLLECTIONS: │\n│ ┌──────────────────────────────────────────────────────────────┐ │\n│ │ │ │\n│ │ 1. SIZE: empty, one, many, boundary (MAX_INT) │ │\n│ │ 2. CONTENT: expected, unexpected, duplicates, nulls │ │\n│ │ 3. ORDER: insertion, sorted, reversed, random │ │\n│ │ 4. MUTATION: add, remove, update, clear │ │\n│ │ 5. THREAD SAFETY: concurrent reads, writes, iteration │ │\n│ │ 6. MEMORY: growth, leaks, large collections │ │\n│ │ │ │\n│ └──────────────────────────────────────────────────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Deep Dive: AssertJ Collection Assertions\n\n### Basic Assertions\n\n```java\nimport static org.assertj.core.api.Assertions.*;\n\n// Size assertions\nassertThat(list).isEmpty();\nassertThat(list).isNotEmpty();\nassertThat(list).hasSize(3);\nassertThat(list).hasSizeGreaterThan(2);\nassertThat(list).hasSizeLessThanOrEqualTo(10);\nassertThat(list).hasSameSizeAs(otherList);\n\n// Null assertions\nassertThat(list).isNull();\nassertThat(list).isNotNull();\nassertThat(list).isNullOrEmpty();\n```\n\n### Content Assertions\n\n```java\nList colors = List.of(\"red\", \"green\", \"blue\");\n\n// Contains (order doesn't matter, can have extras)\nassertThat(colors).contains(\"red\");\nassertThat(colors).contains(\"red\", \"green\");\nassertThat(colors).containsOnly(\"red\", \"green\", \"blue\"); // Only these, any order\nassertThat(colors).containsExactly(\"red\", \"green\", \"blue\"); // Exact order!\nassertThat(colors).containsExactlyInAnyOrder(\"blue\", \"red\", \"green\"); // Any order\n\n// Does not contain\nassertThat(colors).doesNotContain(\"yellow\");\nassertThat(colors).doesNotContainNull();\n\n// Contains duplicates\nassertThat(List.of(\"a\", \"a\", \"b\")).containsDuplicates();\nassertThat(List.of(\"a\", \"b\", \"c\")).doesNotHaveDuplicates();\n\n// Start/end\nassertThat(colors).startsWith(\"red\");\nassertThat(colors).endsWith(\"blue\");\nassertThat(colors).containsSequence(\"green\", \"blue\");\n```\n\n### Condition-Based Assertions\n\n```java\nList users = getUsers();\n\n// All match\nassertThat(users).allMatch(u -> u.getAge() >= 18);\nassertThat(users).allSatisfy(u -> {\n assertThat(u.getName()).isNotBlank();\n assertThat(u.getEmail()).contains(\"@\");\n});\n\n// Any match\nassertThat(users).anyMatch(u -> u.isAdmin());\nassertThat(users).anySatisfy(u -> assertThat(u.getRole()).isEqualTo(\"ADMIN\"));\n\n// None match\nassertThat(users).noneMatch(u -> u.getAge() < 0);\n\n// Single element matching\nassertThat(users).singleElement()\n .satisfies(u -> assertThat(u.getName()).isEqualTo(\"Alice\"));\n```\n\n### Extracting Properties\n\n```java\nList users = List.of(\n new User(\"Alice\", 30),\n new User(\"Bob\", 25),\n new User(\"Charlie\", 35)\n);\n\n// Extract single property\nassertThat(users)\n .extracting(User::getName)\n .containsExactly(\"Alice\", \"Bob\", \"Charlie\");\n\n// Extract multiple properties\nassertThat(users)\n .extracting(User::getName, User::getAge)\n .containsExactly(\n tuple(\"Alice\", 30),\n tuple(\"Bob\", 25),\n tuple(\"Charlie\", 35)\n );\n\n// Flat extracting (for nested collections)\nassertThat(orders)\n .flatExtracting(Order::getItems)\n .extracting(Item::getSku)\n .contains(\"SKU-001\", \"SKU-002\");\n```\n\n### Filtering\n\n```java\nList users = getUsers();\n\n// Filter then assert\nassertThat(users)\n .filteredOn(u -> u.getAge() >= 18)\n .hasSize(5)\n .allMatch(User::isAdult);\n\n// Filter by property\nassertThat(users)\n .filteredOn(\"role\", \"ADMIN\")\n .hasSize(2);\n\n// Filter on condition\nassertThat(users)\n .filteredOn(u -> u.isActive())\n .extracting(User::getName)\n .containsOnly(\"Alice\", \"Bob\");\n```\n\n### Map Assertions\n\n```java\nMap scores = Map.of(\"Alice\", 100, \"Bob\", 95);\n\n// Basic\nassertThat(scores).isEmpty();\nassertThat(scores).isNotEmpty();\nassertThat(scores).hasSize(2);\n\n// Keys\nassertThat(scores).containsKey(\"Alice\");\nassertThat(scores).containsKeys(\"Alice\", \"Bob\");\nassertThat(scores).containsOnlyKeys(\"Alice\", \"Bob\");\nassertThat(scores).doesNotContainKey(\"Charlie\");\n\n// Values\nassertThat(scores).containsValue(100);\nassertThat(scores).containsValues(100, 95);\n\n// Entries\nassertThat(scores).containsEntry(\"Alice\", 100);\nassertThat(scores).containsExactly(\n entry(\"Alice\", 100),\n entry(\"Bob\", 95)\n);\n\n// Extract from entries\nassertThat(scores)\n .extractingFromEntries(Map.Entry::getKey)\n .contains(\"Alice\", \"Bob\");\n```\n\n---\n\n## Deep Dive: Hamcrest Matchers\n\n### Collection Matchers\n\n```java\nimport static org.hamcrest.MatcherAssert.assertThat;\nimport static org.hamcrest.Matchers.*;\n\nList colors = List.of(\"red\", \"green\", \"blue\");\n\n// Size\nassertThat(colors, hasSize(3));\nassertThat(colors, empty());\nassertThat(colors, not(empty()));\n\n// Contains\nassertThat(colors, hasItem(\"red\"));\nassertThat(colors, hasItems(\"red\", \"green\"));\nassertThat(colors, contains(\"red\", \"green\", \"blue\")); // Exact order\nassertThat(colors, containsInAnyOrder(\"blue\", \"red\", \"green\"));\n\n// Every item\nassertThat(colors, everyItem(not(emptyString())));\nassertThat(colors, everyItem(containsString(\"e\"))); // All contain 'e'? No!\n```\n\n### Map Matchers\n\n```java\nMap scores = Map.of(\"Alice\", 100, \"Bob\", 95);\n\nassertThat(scores, hasKey(\"Alice\"));\nassertThat(scores, hasValue(100));\nassertThat(scores, hasEntry(\"Alice\", 100));\nassertThat(scores, aMapWithSize(2));\n```\n\n### Combining Matchers\n\n```java\n// allOf - AND\nassertThat(colors, allOf(\n hasSize(3),\n hasItem(\"red\"),\n not(hasItem(\"yellow\"))\n));\n\n// anyOf - OR\nassertThat(colors, anyOf(\n hasItem(\"red\"),\n hasItem(\"yellow\")\n));\n\n// both/either\nassertThat(colors, both(hasItem(\"red\")).and(hasSize(3)));\nassertThat(colors, either(empty()).or(hasItem(\"red\")));\n```\n\n---\n\n## Deep Dive: Debugging Collection Issues\n\n### ConcurrentModificationException\n\n```java\n// ═══════════════════════════════════════════════════════════════\n// CAUSE 1: Modifying during for-each\n// ═══════════════════════════════════════════════════════════════\nList items = new ArrayList<>(List.of(\"a\", \"b\", \"c\"));\nfor (String item : items) {\n if (item.equals(\"b\")) {\n items.remove(item); // ConcurrentModificationException!\n }\n}\n\n// Fix 1: Use Iterator.remove()\nIterator it = items.iterator();\nwhile (it.hasNext()) {\n if (it.next().equals(\"b\")) {\n it.remove(); // Safe!\n }\n}\n\n// Fix 2: Use removeIf()\nitems.removeIf(item -> item.equals(\"b\"));\n\n// Fix 3: Collect then remove\nList toRemove = items.stream()\n .filter(item -> item.equals(\"b\"))\n .toList();\nitems.removeAll(toRemove);\n\n// ═══════════════════════════════════════════════════════════════\n// CAUSE 2: Multi-threaded access\n// ═══════════════════════════════════════════════════════════════\n// Thread 1\nfor (String item : sharedList) { // Iterating\n process(item);\n}\n// Thread 2\nsharedList.add(\"new\"); // Concurrent modification!\n\n// Fix: Use ConcurrentHashMap, CopyOnWriteArrayList, or explicit synchronization\n```\n\n### Memory Leak Detection\n\n```java\n// ═══════════════════════════════════════════════════════════════\n// COMMON LEAK: Static collection growing forever\n// ═══════════════════════════════════════════════════════════════\npublic class LeakyCache {\n // ❌ Static map that only grows\n private static final Map cache = new HashMap<>();\n\n public void process(String key, byte[] data) {\n cache.put(key, data); // Never removed!\n }\n}\n\n// ═══════════════════════════════════════════════════════════════\n// DETECTION: Using VisualVM or JFR\n// ═══════════════════════════════════════════════════════════════\n// 1. Take heap dump\n// jcmd GC.heap_dump /tmp/heap.hprof\n\n// 2. Analyze in VisualVM or IntelliJ\n// Look for: Collections with many elements, growing over time\n\n// 3. Or use weak references\nprivate static final Map> cache = new WeakHashMap<>();\n\n// ═══════════════════════════════════════════════════════════════\n// TEST FOR MEMORY LEAK\n// ═══════════════════════════════════════════════════════════════\n@Test\nvoid shouldNotLeakMemory() {\n Runtime runtime = Runtime.getRuntime();\n long beforeMemory = runtime.totalMemory() - runtime.freeMemory();\n\n for (int i = 0; i < 100_000; i++) {\n service.process(\"key-\" + i, new byte[1024]);\n }\n\n System.gc();\n Thread.sleep(1000);\n\n long afterMemory = runtime.totalMemory() - runtime.freeMemory();\n long leaked = afterMemory - beforeMemory;\n\n assertThat(leaked).isLessThan(50 * 1024 * 1024); // Less than 50MB\n}\n```\n\n### Debugging with IntelliJ\n\n```java\n// ═══════════════════════════════════════════════════════════════\n// TIP 1: Stream Trace (Debug menu when stopped at stream)\n// ═══════════════════════════════════════════════════════════════\nList result = items.stream()\n .filter(s -> s.length() > 3) // Set breakpoint here\n .map(String::toUpperCase)\n .collect(toList());\n// Click \"Trace Current Stream Chain\" in debugger\n\n// ═══════════════════════════════════════════════════════════════\n// TIP 2: Collection renderer customization\n// ═══════════════════════════════════════════════════════════════\n// Settings → Build, Execution, Deployment → Debugger → Data Views\n// Add custom renderer for your types\n\n// ═══════════════════════════════════════════════════════════════\n// TIP 3: Conditional breakpoints on collection state\n// ═══════════════════════════════════════════════════════════════\n// Right-click breakpoint → Condition:\n// items.size() > 100 || items.contains(\"suspicious\")\n\n// ═══════════════════════════════════════════════════════════════\n// TIP 4: Watch expressions for collections\n// ═══════════════════════════════════════════════════════════════\n// Add to watches:\n// items.stream().filter(s -> s.startsWith(\"ERROR\")).count()\n// map.entrySet().stream().filter(e -> e.getValue() == null).toList()\n```\n\n---\n\n## Deep Dive: Testing Thread Safety\n\n### Basic Concurrent Test\n\n```java\n@Test\nvoid shouldHandleConcurrentAccess() throws InterruptedException {\n ConcurrentMap map = new ConcurrentHashMap<>();\n int threadCount = 10;\n int operationsPerThread = 1000;\n CountDownLatch latch = new CountDownLatch(threadCount);\n\n for (int t = 0; t < threadCount; t++) {\n new Thread(() -> {\n for (int i = 0; i < operationsPerThread; i++) {\n map.merge(\"counter\", 1, Integer::sum);\n }\n latch.countDown();\n }).start();\n }\n\n latch.await(10, TimeUnit.SECONDS);\n\n assertThat(map.get(\"counter\"))\n .isEqualTo(threadCount * operationsPerThread);\n}\n```\n\n### Testing for Race Conditions\n\n```java\n@Test\n@RepeatedTest(100) // Run many times to catch intermittent issues\nvoid shouldBeThreadSafe() throws Exception {\n var service = new InventoryService();\n service.setStock(\"item\", 100);\n\n int threadCount = 10;\n int reservationsPerThread = 10;\n ExecutorService executor = Executors.newFixedThreadPool(threadCount);\n CountDownLatch startLatch = new CountDownLatch(1);\n CountDownLatch doneLatch = new CountDownLatch(threadCount);\n AtomicInteger successCount = new AtomicInteger(0);\n\n for (int i = 0; i < threadCount; i++) {\n executor.submit(() -> {\n try {\n startLatch.await(); // All threads start together\n for (int j = 0; j < reservationsPerThread; j++) {\n if (service.reserve(\"item\", 1)) {\n successCount.incrementAndGet();\n }\n }\n } catch (InterruptedException e) {\n Thread.currentThread().interrupt();\n } finally {\n doneLatch.countDown();\n }\n });\n }\n\n startLatch.countDown(); // GO!\n doneLatch.await(10, TimeUnit.SECONDS);\n executor.shutdown();\n\n // Exactly 100 reservations should succeed (starting stock)\n assertThat(successCount.get()).isEqualTo(100);\n assertThat(service.getStock(\"item\")).isZero();\n}\n```\n\n### Using JCStress (for serious concurrency testing)\n\n```java\n// JCStress is a specialized framework for testing concurrency\n// See: https://github.com/openjdk/jcstress\n\n@JCStressTest\n@Outcome(id = \"1, 1\", expect = Expect.ACCEPTABLE)\n@Outcome(id = \"0, 0\", expect = Expect.FORBIDDEN)\n@State\npublic class MapRaceTest {\n private final Map map = new HashMap<>();\n\n @Actor\n public void actor1(II_Result r) {\n map.put(\"key\", 1);\n r.r1 = map.get(\"key\");\n }\n\n @Actor\n public void actor2(II_Result r) {\n map.put(\"key\", 2);\n r.r2 = map.get(\"key\");\n }\n}\n```\n\n---\n\n## Deep Dive: Testing Edge Cases\n\n### Null Handling\n\n```java\n@Test\nvoid shouldHandleNullElements() {\n List withNulls = Arrays.asList(\"a\", null, \"c\");\n\n // Test null filtering\n List filtered = service.filterNulls(withNulls);\n assertThat(filtered)\n .doesNotContainNull()\n .containsExactly(\"a\", \"c\");\n}\n\n@Test\nvoid shouldRejectNullInput() {\n assertThatThrownBy(() -> service.process(null))\n .isInstanceOf(NullPointerException.class)\n .hasMessageContaining(\"input must not be null\");\n}\n\n@Test\nvoid shouldHandleCollectionWithNulls() {\n List input = Arrays.asList(\"a\", null, \"b\");\n\n assertThatThrownBy(() -> Set.copyOf(input))\n .isInstanceOf(NullPointerException.class);\n}\n```\n\n### Empty Collection Handling\n\n```java\n@Test\nvoid shouldHandleEmptyCollection() {\n List empty = List.of();\n\n assertThat(service.findMax(empty)).isEmpty();\n assertThat(service.findMin(empty)).isEmpty();\n assertThat(service.average(empty)).isNaN();\n}\n\n@Test\nvoid shouldReturnEmptyNotNull() {\n List users = userService.findByRole(\"NONEXISTENT\");\n\n assertThat(users)\n .isNotNull()\n .isEmpty();\n}\n```\n\n### Boundary Conditions\n\n```java\n@Test\nvoid shouldHandleSingleElement() {\n List single = List.of(42);\n\n assertThat(service.findMax(single)).hasValue(42);\n assertThat(service.findMin(single)).hasValue(42);\n assertThat(service.sort(single)).containsExactly(42);\n}\n\n@Test\nvoid shouldHandleLargeCollection() {\n List large = IntStream.range(0, 1_000_000)\n .boxed()\n .collect(Collectors.toList());\n\n // Should complete in reasonable time\n assertTimeout(Duration.ofSeconds(5), () -> {\n service.process(large);\n });\n}\n\n@Test\nvoid shouldHandleDuplicates() {\n List withDupes = List.of(\"a\", \"a\", \"b\", \"b\", \"b\");\n\n Set unique = service.toUniqueSet(withDupes);\n assertThat(unique).containsExactlyInAnyOrder(\"a\", \"b\");\n\n Map counts = service.countOccurrences(withDupes);\n assertThat(counts)\n .containsEntry(\"a\", 2L)\n .containsEntry(\"b\", 3L);\n}\n```\n\n### Order Verification\n\n```java\n@Test\nvoid shouldMaintainInsertionOrder() {\n SequencedSet set = new LinkedHashSet<>();\n set.add(\"first\");\n set.add(\"second\");\n set.add(\"third\");\n\n assertThat(set).containsExactly(\"first\", \"second\", \"third\");\n assertThat(set.getFirst()).isEqualTo(\"first\");\n assertThat(set.getLast()).isEqualTo(\"third\");\n}\n\n@Test\nvoid shouldSortCorrectly() {\n List users = List.of(\n new User(\"Charlie\", 30),\n new User(\"Alice\", 25),\n new User(\"Bob\", 35)\n );\n\n List sorted = service.sortByName(users);\n\n assertThat(sorted)\n .extracting(User::getName)\n .containsExactly(\"Alice\", \"Bob\", \"Charlie\");\n}\n\n@Test\nvoid shouldReverseCorrectly() {\n List original = List.of(1, 2, 3, 4, 5);\n List reversed = service.reverse(original);\n\n assertThat(reversed).containsExactly(5, 4, 3, 2, 1);\n assertThat(original).containsExactly(1, 2, 3, 4, 5); // Original unchanged\n}\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Not Testing Exact Content\n\n```java\n// ❌ WRONG: Passes even with unexpected elements\n@Test\nvoid testBad() {\n Set result = service.getActiveUsers();\n assertTrue(result.contains(\"Alice\"));\n assertTrue(result.contains(\"Bob\"));\n // What if result also contains \"Hacker\"?\n}\n\n// ✅ CORRECT: Verify exact content\n@Test\nvoid testGood() {\n Set result = service.getActiveUsers();\n assertThat(result).containsExactlyInAnyOrder(\"Alice\", \"Bob\");\n}\n```\n\n### Mistake 2: Order Sensitivity in Set Tests\n\n```java\n// ❌ WRONG: Sets don't guarantee order\n@Test\nvoid testBad() {\n Set result = service.getTags();\n assertThat(result).containsExactly(\"a\", \"b\", \"c\");\n // Fails randomly depending on hash order!\n}\n\n// ✅ CORRECT: Use order-insensitive assertion\n@Test\nvoid testGood() {\n Set result = service.getTags();\n assertThat(result).containsExactlyInAnyOrder(\"a\", \"b\", \"c\");\n}\n```\n\n### Mistake 3: Shallow Collection Comparison\n\n```java\n// ❌ WRONG: Only checks references, not content\n@Test\nvoid testBad() {\n List result = service.getUsers();\n User expected = new User(\"Alice\", 25);\n assertTrue(result.contains(expected));\n // Fails if User doesn't implement equals()!\n}\n\n// ✅ CORRECT: Use field extractors\n@Test\nvoid testGood() {\n List result = service.getUsers();\n assertThat(result)\n .extracting(User::getName, User::getAge)\n .contains(tuple(\"Alice\", 25));\n}\n```\n\n### Mistake 4: Not Testing Immutability\n\n```java\n// ❌ WRONG: Doesn't verify immutability\n@Test\nvoid testBad() {\n List result = service.getImmutableList();\n assertThat(result).containsExactly(\"a\", \"b\");\n // But is it actually immutable?\n}\n\n// ✅ CORRECT: Verify modification throws\n@Test\nvoid testGood() {\n List result = service.getImmutableList();\n assertThat(result).containsExactly(\"a\", \"b\");\n\n assertThatThrownBy(() -> result.add(\"c\"))\n .isInstanceOf(UnsupportedOperationException.class);\n}\n```\n\n### Mistake 5: Flaky Concurrent Tests\n\n```java\n// ❌ WRONG: Race condition in test itself\n@Test\nvoid testBad() {\n var service = new MyService();\n new Thread(() -> service.addItem(\"a\")).start();\n new Thread(() -> service.addItem(\"b\")).start();\n\n Thread.sleep(100); // Hope threads finished!\n assertThat(service.getItems()).hasSize(2);\n}\n\n// ✅ CORRECT: Use proper synchronization\n@Test\nvoid testGood() throws InterruptedException {\n var service = new MyService();\n var latch = new CountDownLatch(2);\n\n new Thread(() -> { service.addItem(\"a\"); latch.countDown(); }).start();\n new Thread(() -> { service.addItem(\"b\"); latch.countDown(); }).start();\n\n assertTrue(latch.await(5, TimeUnit.SECONDS));\n assertThat(service.getItems()).hasSize(2);\n}\n```\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Collection Test Suite\n\nWrite comprehensive tests for this method:\n\n```java\npublic class ListUtils {\n public static List removeDuplicates(List input) {\n return new ArrayList<>(new LinkedHashSet<>(input));\n }\n}\n```\n\n**✅ Solution:**\n\n```java\nclass ListUtilsTest {\n\n @Test\n void shouldRemoveDuplicatesPreservingOrder() {\n List input = List.of(\"a\", \"b\", \"a\", \"c\", \"b\");\n List result = ListUtils.removeDuplicates(input);\n assertThat(result).containsExactly(\"a\", \"b\", \"c\");\n }\n\n @Test\n void shouldHandleEmptyList() {\n List result = ListUtils.removeDuplicates(List.of());\n assertThat(result).isEmpty();\n }\n\n @Test\n void shouldHandleSingleElement() {\n List result = ListUtils.removeDuplicates(List.of(\"only\"));\n assertThat(result).containsExactly(\"only\");\n }\n\n @Test\n void shouldHandleNoDuplicates() {\n List input = List.of(1, 2, 3);\n List result = ListUtils.removeDuplicates(input);\n assertThat(result).containsExactly(1, 2, 3);\n }\n\n @Test\n void shouldHandleAllDuplicates() {\n List input = List.of(\"x\", \"x\", \"x\");\n List result = ListUtils.removeDuplicates(input);\n assertThat(result).containsExactly(\"x\");\n }\n\n @Test\n void shouldHandleNullInList() {\n List input = Arrays.asList(\"a\", null, \"b\", null);\n List result = ListUtils.removeDuplicates(input);\n assertThat(result).containsExactly(\"a\", null, \"b\");\n }\n\n @Test\n void shouldReturnMutableList() {\n List result = ListUtils.removeDuplicates(List.of(\"a\"));\n assertThatCode(() -> result.add(\"b\")).doesNotThrowAnyException();\n }\n\n @Test\n void shouldNotModifyOriginal() {\n List original = new ArrayList<>(List.of(\"a\", \"b\", \"a\"));\n ListUtils.removeDuplicates(original);\n assertThat(original).containsExactly(\"a\", \"b\", \"a\");\n }\n}\n```\n\n### Exercise 2: Debug the Race Condition\n\nFix this thread-unsafe code and write a test that catches the bug:\n\n```java\npublic class Counter {\n private Map counts = new HashMap<>();\n\n public void increment(String key) {\n Integer current = counts.get(key);\n counts.put(key, current == null ? 1 : current + 1);\n }\n\n public int get(String key) {\n return counts.getOrDefault(key, 0);\n }\n}\n```\n\n**✅ Solution:**\n\n```java\n// Fixed implementation\npublic class Counter {\n private final ConcurrentMap counts = new ConcurrentHashMap<>();\n\n public void increment(String key) {\n counts.computeIfAbsent(key, k -> new AtomicInteger(0))\n .incrementAndGet();\n }\n\n public int get(String key) {\n AtomicInteger counter = counts.get(key);\n return counter == null ? 0 : counter.get();\n }\n}\n\n// Test that catches the race condition\n@Test\n@RepeatedTest(10)\nvoid shouldIncrementAtomically() throws InterruptedException {\n Counter counter = new Counter();\n int threadCount = 100;\n int incrementsPerThread = 1000;\n ExecutorService executor = Executors.newFixedThreadPool(threadCount);\n CountDownLatch latch = new CountDownLatch(threadCount);\n\n for (int i = 0; i < threadCount; i++) {\n executor.submit(() -> {\n for (int j = 0; j < incrementsPerThread; j++) {\n counter.increment(\"key\");\n }\n latch.countDown();\n });\n }\n\n latch.await(30, TimeUnit.SECONDS);\n executor.shutdown();\n\n assertThat(counter.get(\"key\"))\n .isEqualTo(threadCount * incrementsPerThread); // 100,000\n}\n```\n\n### Exercise 3: Memory Leak Test\n\nWrite a test that detects if a cache is leaking memory:\n\n```java\npublic class SimpleCache {\n private final Map cache = new HashMap<>();\n private final int maxSize;\n\n public SimpleCache(int maxSize) {\n this.maxSize = maxSize;\n }\n\n public void put(K key, V value) {\n if (cache.size() >= maxSize) {\n // Should evict old entries, but doesn't!\n }\n cache.put(key, value);\n }\n\n public V get(K key) {\n return cache.get(key);\n }\n}\n```\n\n**✅ Solution:**\n\n```java\n@Test\nvoid shouldNotExceedMaxSize() {\n SimpleCache cache = new SimpleCache<>(100);\n\n // Add more than maxSize entries\n for (int i = 0; i < 1000; i++) {\n cache.put(i, new byte[1024]); // 1KB each\n }\n\n // If leaking, we'll have 1000 entries (~1MB)\n // If working, we'll have 100 entries (~100KB)\n\n // Test via reflection or add a size() method\n assertThat(getCacheSize(cache)).isLessThanOrEqualTo(100);\n}\n\n@Test\nvoid shouldNotLeakMemory() {\n SimpleCache cache = new SimpleCache<>(100);\n\n // Get baseline memory\n System.gc();\n long baseline = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();\n\n // Add entries that should be evicted\n for (int i = 0; i < 10_000; i++) {\n cache.put(i, new byte[10_000]); // 10KB each\n }\n\n System.gc();\n Thread.sleep(100);\n\n long after = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();\n long used = after - baseline;\n\n // If working correctly: ~100 entries * 10KB = ~1MB\n // If leaking: 10000 entries * 10KB = ~100MB\n assertThat(used).isLessThan(10_000_000); // Less than 10MB\n}\n\n// Fixed cache\npublic class FixedCache {\n private final Map cache = new LinkedHashMap<>() {\n @Override\n protected boolean removeEldestEntry(Map.Entry eldest) {\n return size() > maxSize;\n }\n };\n private final int maxSize;\n\n // ... rest of implementation\n}\n```\n\n---\n\n## 🎤 Senior-Level Interview Questions\n\n### Question 1: contains() vs containsExactly()\n\n**Q: What's the difference between AssertJ's contains() and containsExactly()?**\n\n**A:**\n\n```java\nList list = List.of(\"a\", \"b\", \"c\");\n\n// contains: Element exists, allows others, any order\nassertThat(list).contains(\"a\"); // ✓\nassertThat(list).contains(\"a\", \"b\"); // ✓ (subset OK)\nassertThat(list).contains(\"b\", \"a\"); // ✓ (any order)\n\n// containsExactly: Exact elements in exact order\nassertThat(list).containsExactly(\"a\", \"b\", \"c\"); // ✓\nassertThat(list).containsExactly(\"a\", \"b\"); // ✗ (missing c)\nassertThat(list).containsExactly(\"c\", \"b\", \"a\"); // ✗ (wrong order)\n\n// containsExactlyInAnyOrder: Exact elements, any order\nassertThat(list).containsExactlyInAnyOrder(\"c\", \"b\", \"a\"); // ✓\n```\n\n### Question 2: Testing Thread Safety\n\n**Q: How do you test that a collection implementation is thread-safe?**\n\n**A:**\n\n1. **Multiple threads hitting same operation**:\n```java\nExecutorService executor = Executors.newFixedThreadPool(10);\nCountDownLatch latch = new CountDownLatch(10);\n\nfor (int i = 0; i < 10; i++) {\n executor.submit(() -> {\n for (int j = 0; j < 1000; j++) {\n collection.add(\"item-\" + j);\n }\n latch.countDown();\n });\n}\n\nlatch.await();\n// Verify final state is consistent\n```\n\n2. **Race condition tests with CountDownLatch**:\n```java\nCountDownLatch startGate = new CountDownLatch(1);\n// All threads wait at gate, then start simultaneously\n```\n\n3. **JCStress** for thorough analysis\n4. **@RepeatedTest** to catch intermittent failures\n\n### Question 3: Debugging ConcurrentModificationException\n\n**Q: How do you debug a ConcurrentModificationException that only happens in production?**\n\n**A:**\n\n1. **Add logging around collection operations**:\n```java\nlog.debug(\"Before iteration, size={}\", list.size());\nfor (Item item : list) {\n log.trace(\"Processing {}\", item);\n // ...\n}\n```\n\n2. **Check for multi-threaded access**:\n```java\n// Temporarily wrap with checked wrapper\nList checked = Collections.checkedList(\n Collections.synchronizedList(list), String.class);\n```\n\n3. **Use stack traces from the exception**:\n- The exception includes both the iterator creation and modification point\n- Compare thread IDs in logs around those times\n\n4. **Consider using CopyOnWriteArrayList temporarily** to see if problem disappears\n\n### Question 4: Memory Leak Detection\n\n**Q: How would you detect if a Map is causing a memory leak?**\n\n**A:**\n\n1. **Heap dump analysis**:\n```bash\njcmd GC.heap_dump /tmp/heap.hprof\n# Open in VisualVM or IntelliJ\n```\n\n2. **JFR recording**:\n```bash\njcmd JFR.start name=leak duration=60s filename=/tmp/leak.jfr\n```\n\n3. **Test with assertions**:\n```java\n@Test\nvoid mapShouldNotGrowUnbounded() {\n for (int i = 0; i < 100_000; i++) {\n service.process(\"key-\" + i);\n }\n assertThat(service.getCacheSize()).isLessThan(1000);\n}\n```\n\n4. **WeakHashMap test**:\n```java\n// If using WeakHashMap, test that entries can be GC'd\nmap.put(key, value);\nkey = null;\nSystem.gc();\nassertThat(map).isEmpty(); // Key was weak-referenced\n```\n\n### Question 5: Test Isolation\n\n**Q: How do you ensure tests don't affect each other when testing shared collections?**\n\n**A:**\n\n1. **Create fresh instances**:\n```java\n@BeforeEach\nvoid setUp() {\n collection = new ArrayList<>();\n}\n```\n\n2. **Use defensive copies**:\n```java\nprivate static final List TEST_DATA = List.of(\"a\", \"b\", \"c\");\n\n@Test\nvoid test1() {\n List data = new ArrayList<>(TEST_DATA); // Copy\n // Modify data safely\n}\n```\n\n3. **Avoid static shared state**\n4. **Clear collections in @AfterEach if reusing**\n5. **Use @DirtiesContext in Spring if needed**\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### AssertJ Essentials\n\n| Assertion | Use Case |\n|-----------|----------|\n| `contains()` | Element exists, others OK |\n| `containsExactly()` | Exact elements, exact order |\n| `containsExactlyInAnyOrder()` | Exact elements, any order |\n| `extracting()` | Test object properties |\n| `filteredOn()` | Filter then assert |\n\n### Test Coverage Areas\n\n1. **Size**: Empty, one, many, boundary\n2. **Content**: Expected, unexpected, duplicates, nulls\n3. **Order**: Insertion, sorted, reversed\n4. **Mutation**: Add, remove, update, clear\n5. **Thread safety**: Concurrent operations\n6. **Memory**: Growth, leaks\n\n### Debugging Tools\n\n- IntelliJ Stream Trace\n- Conditional breakpoints\n- JFR and heap dumps\n- WeakHashMap for leak tests\n\n---\n\n## 🏁 Conclusion\n\nEffective collection testing requires comprehensive coverage of edge cases and clear assertions. The key insights are:\n\n1. **Use assertThat(list).containsExactly()** for precise verification\n2. **Test edge cases**: empty, single, large, with nulls\n3. **Verify immutability** by attempting modification\n4. **Use proper synchronization** in concurrent tests\n5. **Monitor memory** for leak detection\n\nIn the next article, we'll explore Performance and Production Pitfalls - real-world issues and how to avoid them.\n\n---\n\n## 📅 Review Schedule\n\nTo solidify your understanding:\n- **Tomorrow**: Rewrite one test using AssertJ\n- **In 3 days**: Add edge case tests to existing code\n- **In 1 week**: Run a concurrent test repeatedly\n- **In 2 weeks**: Analyze heap dump for a production app\n\n---\n\n**Next in Series**: [Part 28: Performance, Memory & Production Pitfalls](/articles/java-collections-part-28-performance-pitfalls)\n\n**Series Navigation**: [Part 0: How to Use This Series](/articles/java-collections-part-0-how-to-use-this-series) | [Full Index](/articles/java-collections-compendium)\n","2026-06-10T07:19:26.199243Z","2026-06-10T07:19:26.199996Z",15,"Java",5,27,{"_33":57,"_96":59,"_97":98,"_99":100,"_101":-5,"_102":-5,"_103":-5,"_104":-5,"_105":-5},{"_33":115,"_74":114,"_37":136,"_77":140,"_95":-5},[124,125,126,127,128],{"_33":115,"_74":121,"_37":122,"_77":123,"_79":24},"Java Collections Compendium: From Fundamentals to Production Mastery","java-collections-compendium","A comprehensive 30-part series covering everything about Java Collections Framework. From foundational concepts like generics and equals/hashCode contract, through deep dives into ArrayList, LinkedList, HashMap, TreeMap, ConcurrentHashMap, to modern features like immutable collections, Java 21 sequenced collections, streams & collectors, and production patterns with JPA, Jackson & Spring. Includes real production stories, mental models, exercises, and 50+ interview questions.",{"_33":137,"_74":138,"_37":139},{"_33":89,"_74":90,"_37":91},{"_33":135,"_74":114,"_37":136},{"_33":132,"_74":133,"_37":134},{"_33":129,"_74":130,"_37":131},379,"Testing","testing",380,"Junit","junit",14,"java",381,"Assertj","assertj","Category for Java"]