[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6},"actionData","errors","routes/blog-tag",{"_7":8,"_9":10},"tag",{"_33":83,"_74":84,"_37":85},"articles",{"_11":12,"_13":14,"_15":16,"_17":18,"_19":20,"_21":22,"_23":24,"_25":24,"_26":16,"_27":28},"content",[31,32],"pageable",{"_29":22,"_30":20},"totalElements",2,"totalPages",1,"size",10,"number",0,"first",true,"last","numberOfElements","empty",false,"page_number","page_size",{"_33":106,"_35":107,"_37":108,"_39":109,"_11":110,"_42":-5,"_43":24,"_44":28,"_45":46,"_47":111,"_49":112,"_51":52,"_53":54,"_55":54,"_56":57,"_58":59,"_60":61,"_62":54,"_63":57,"_64":113,"_66":114,"_68":115,"_70":116,"_72":117},{"_33":34,"_35":36,"_37":38,"_39":40,"_11":41,"_42":-5,"_43":24,"_44":28,"_45":46,"_47":48,"_49":50,"_51":52,"_53":54,"_55":54,"_56":57,"_58":59,"_60":61,"_62":54,"_63":57,"_64":65,"_66":67,"_68":69,"_70":71,"_72":73},"id",61,"title","Logging & Observability","slug","docker-part-11-logging","excerpt","Log drivers, structured logging, health checks","# Docker Compendium Part 11: Logging & Observability\n\n\"Where are my logs?\" - the most common question when debugging containers. Docker handles logging differently than traditional deployments. This article covers logging drivers, structured logging, metrics, health checks, and building observable containerized applications.\n\n## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| Topic | Logging drivers, structured logging, health checks, metrics |\n| Complexity | Intermediate |\n| Prerequisites | Basic Docker usage, Part 1 (Container Internals) |\n| Key Insight | Containers should log to stdout/stderr - Docker handles the rest |\n| Time to Master | 2-3 hours |\n\n---\n\n## 🎯 What You'll Learn\n\n- **Logging drivers** - json-file, syslog, fluentd, and when to use each\n- **Structured logging** - JSON logs for machine parsing\n- **Health checks** - liveness vs readiness, implementing correctly\n- **Metrics collection** - Prometheus patterns for containers\n- **Debugging without logs** - when logs aren't enough\n\n---\n\n## 🔥 Production Story: The Silent Failure\n\nAn application ran fine for weeks, then started dropping requests. No errors in logs. The team added more replicas, but problems persisted.\n\n**Investigation:**\n\n```bash\n$ docker logs api-server --tail 100\n# ... normal request logs ...\n# Nothing unusual\n\n$ docker inspect api-server --format '{{.State.Health}}'\n# \"healthy\"\n```\n\nEverything looked fine. But checking deeper:\n\n```bash\n$ curl localhost:8080/health\n{\"status\": \"healthy\"} # HTTP 200\n\n$ curl localhost:8080/api/users\n# ... 30 second timeout ...\n```\n\n**Root cause:** The health check only verified the HTTP server was responding. The database connection pool was exhausted - app could respond to `/health` but not process actual requests.\n\n**The fix:**\n\n```dockerfile\nHEALTHCHECK --interval=30s --timeout=10s --retries=3 \\\n CMD curl -f http://localhost:8080/health/deep || exit 1\n```\n\n```javascript\n// Deep health check that verifies dependencies\napp.get('/health/deep', async (req, res) => {\n try {\n await db.query('SELECT 1');\n await redis.ping();\n res.json({ status: 'healthy', db: 'ok', cache: 'ok' });\n } catch (err) {\n res.status(503).json({ status: 'unhealthy', error: err.message });\n }\n});\n```\n\n**Lesson:** Health checks must verify actual functionality, not just \"is the process running.\"\n\n---\n\n## 🧠 Mental Model: Container Observability Stack\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│ CONTAINER OBSERVABILITY │\n│ │\n│ ┌──────────────────────────────────────────────────────────────────────┐\n│ │ APPLICATION │\n│ │ │\n│ │ Log to stdout/stderr ──────────────────────────────────────────┐ │\n│ │ Expose /metrics endpoint ──────────────────────────────────┐ │ │ \n│ │ Implement /health endpoints ─────────────────────────────┐ │ │ │\n│ └───────────────────────────────────────────────────────────┼─┼───┼────┘\n│ │ │ │\n│ ┌───────────────────────────────────────────────────────────┼─┼───┼────┐\n│ │ DOCKER DAEMON │ │ │ │\n│ │ │ │ │ │\n│ │ Logging Driver ◄─────────────────────────────────────────┘ │ │ │\n│ │ (json-file, fluentd, etc.) │ │ │\n│ │ │ │ │ │\n│ │ ├── json-file ──► /var/lib/docker/containers/*/ │ │ │\n│ │ ├── syslog ────► syslog server │ │ │\n│ │ ├── fluentd ───► Fluentd/Fluent Bit │ │ │\n│ │ └── awslogs ───► CloudWatch │ │ │\n│ │ │ │ │\n│ │ Health Check ◄─────────────────────────────────────────────┘ │ │\n│ │ (HEALTHCHECK instruction) │ │\n│ │ │ │ │\n│ │ └── Updates container status │ │\n│ │ │ │\n│ └─────────────────────────────────────────────────────────────────┼────┘\n│ │\n│ ┌─────────────────────────────────────────────────────────────────┼───┐\n│ │ MONITORING SYSTEM │ │\n│ │ │ │\n│ │ Prometheus ◄───────────────────────────────────────────────────┘ │\n│ │ (scrapes /metrics) │\n│ │ │ │\n│ │ └── Grafana (visualization) │\n│ │ │\n│ │ Log Aggregator (ELK, Loki, etc.) │\n│ │ │ │\n│ │ └── Receives logs from driver │\n│ │ │\n│ └─────────────────────────────────────────────────────────────────────┘\n│ │\n└────────────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## 🔬 Deep Dive\n\n### The Twelve-Factor App Logging Principle\n\nContainers follow the 12-factor app methodology for logs:\n\n> **Treat logs as event streams. A twelve-factor app never concerns itself with routing or storage of its output stream.**\n\n**Translation:** Write to stdout/stderr. Let the platform (Docker) handle the rest.\n\n```python\n# ✅ Good - logs to stdout\nimport logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\nlogger.info(\"Request processed\")\n\n# ❌ Bad - logs to file in container\nlogging.basicConfig(filename='/var/log/app.log')\n```\n\n```javascript\n// ✅ Good - console goes to stdout/stderr\nconsole.log('Info message');\nconsole.error('Error message');\n\n// ❌ Bad - file logging in container\nfs.appendFileSync('/var/log/app.log', message);\n```\n\n### Logging Drivers\n\nDocker captures stdout/stderr and sends to a logging driver:\n\n```bash\n# See current driver\ndocker info --format '{{.LoggingDriver}}'\n\n# Run with specific driver\ndocker run --log-driver json-file nginx\ndocker run --log-driver syslog nginx\ndocker run --log-driver fluentd nginx\ndocker run --log-driver none nginx # No logging\n```\n\n**Available drivers:**\n\n| Driver | Description | Use Case |\n|--------|-------------|----------|\n| json-file | JSON files on disk (default) | Development, simple deployments |\n| syslog | System syslog | Traditional infrastructure |\n| journald | systemd journal | Linux with systemd |\n| fluentd | Fluentd collector | Kubernetes, centralized logging |\n| awslogs | AWS CloudWatch | AWS deployments |\n| gcplogs | Google Cloud Logging | GCP deployments |\n| none | Disable logging | When logs handled elsewhere |\n\n**Configure default driver:**\n\n```json\n// /etc/docker/daemon.json\n{\n \"log-driver\": \"json-file\",\n \"log-opts\": {\n \"max-size\": \"10m\",\n \"max-file\": \"3\"\n }\n}\n```\n\n### JSON-File Driver (Default)\n\nMost common for development and simple deployments:\n\n```bash\n# View logs\ndocker logs container_name\ndocker logs -f container_name # Follow\ndocker logs --tail 100 container_name # Last 100 lines\ndocker logs --since 1h container_name # Last hour\n\n# Log file location\nls /var/lib/docker/containers//*-json.log\n\n# Direct file inspection (raw JSON)\ntail -f /var/lib/docker/containers//*-json.log\n```\n\n**Configure log rotation:**\n\n```yaml\n# docker-compose.yml\nservices:\n app:\n image: myapp\n logging:\n driver: json-file\n options:\n max-size: \"10m\" # Max file size\n max-file: \"3\" # Number of files to keep\n compress: \"true\" # Compress rotated files\n```\n\n```bash\n# Docker run equivalent\ndocker run \\\n --log-opt max-size=10m \\\n --log-opt max-file=3 \\\n myapp\n```\n\n### Structured Logging\n\nJSON logs are machine-parseable:\n\n```python\nimport json\nimport sys\nfrom datetime import datetime\n\ndef log(level, message, **extra):\n entry = {\n \"timestamp\": datetime.utcnow().isoformat(),\n \"level\": level,\n \"message\": message,\n **extra\n }\n print(json.dumps(entry), file=sys.stdout, flush=True)\n\n# Usage\nlog(\"INFO\", \"Request processed\",\n request_id=\"abc123\",\n user_id=456,\n duration_ms=125)\n\n# Output:\n# {\"timestamp\":\"2024-01-15T10:30:00.000Z\",\"level\":\"INFO\",\"message\":\"Request processed\",\"request_id\":\"abc123\",\"user_id\":456,\"duration_ms\":125}\n```\n\n```javascript\n// Node.js with pino (fast JSON logger)\nconst pino = require('pino');\nconst logger = pino();\n\nlogger.info({ requestId: 'abc123', userId: 456 }, 'Request processed');\n```\n\n**Benefits of structured logging:**\n- Machine-parseable (Elasticsearch, Loki, etc.)\n- Filterable by fields\n- Aggregatable for metrics\n- Consistent format\n\n### Health Checks\n\nHealth checks tell Docker (and orchestrators) if a container is working:\n\n```dockerfile\n# Simple HTTP check\nHEALTHCHECK --interval=30s --timeout=10s --retries=3 \\\n CMD curl -f http://localhost:8080/health || exit 1\n\n# Command-based check\nHEALTHCHECK --interval=30s --timeout=10s --retries=3 \\\n CMD pg_isready -U postgres || exit 1\n\n# Custom script\nHEALTHCHECK --interval=30s --timeout=10s --retries=3 \\\n CMD /app/healthcheck.sh\n```\n\n**Health check options:**\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--interval` | 30s | Time between checks |\n| `--timeout` | 30s | Max time for check to complete |\n| `--start-period` | 0s | Initialization grace period |\n| `--retries` | 3 | Failures before unhealthy |\n\n**Container health states:**\n- `starting` - In start-period, checks not counted yet\n- `healthy` - Last N checks passed\n- `unhealthy` - Last N checks failed\n\n```bash\n# Check health status\ndocker inspect --format='{{.State.Health.Status}}' container\n\n# Detailed health info\ndocker inspect --format='{{json .State.Health}}' container | jq\n```\n\n**Comprehensive health check pattern:**\n\n```javascript\n// Express.js health endpoints\napp.get('/health', (req, res) => {\n // Quick liveness check\n res.json({ status: 'alive' });\n});\n\napp.get('/health/ready', async (req, res) => {\n // Readiness - can we serve traffic?\n const checks = await Promise.allSettled([\n db.query('SELECT 1'),\n redis.ping(),\n fetch(upstreamService + '/health')\n ]);\n\n const results = {\n database: checks[0].status === 'fulfilled' ? 'ok' : 'fail',\n cache: checks[1].status === 'fulfilled' ? 'ok' : 'fail',\n upstream: checks[2].status === 'fulfilled' ? 'ok' : 'fail'\n };\n\n const healthy = Object.values(results).every(v => v === 'ok');\n res.status(healthy ? 200 : 503).json({\n status: healthy ? 'ready' : 'not_ready',\n checks: results\n });\n});\n```\n\n### Metrics with Prometheus\n\nStandard pattern for container metrics:\n\n```javascript\n// Node.js with prom-client\nconst prometheus = require('prom-client');\n\n// Default metrics (memory, CPU, etc.)\nprometheus.collectDefaultMetrics();\n\n// Custom metrics\nconst httpRequestsTotal = new prometheus.Counter({\n name: 'http_requests_total',\n help: 'Total HTTP requests',\n labelNames: ['method', 'path', 'status']\n});\n\nconst httpRequestDuration = new prometheus.Histogram({\n name: 'http_request_duration_seconds',\n help: 'HTTP request duration',\n labelNames: ['method', 'path'],\n buckets: [0.1, 0.5, 1, 2, 5]\n});\n\n// Middleware to track requests\napp.use((req, res, next) => {\n const end = httpRequestDuration.startTimer();\n res.on('finish', () => {\n httpRequestsTotal.inc({\n method: req.method,\n path: req.route?.path || req.path,\n status: res.statusCode\n });\n end({ method: req.method, path: req.route?.path || req.path });\n });\n next();\n});\n\n// Metrics endpoint\napp.get('/metrics', async (req, res) => {\n res.set('Content-Type', prometheus.register.contentType);\n res.send(await prometheus.register.metrics());\n});\n```\n\n**Prometheus scrape config:**\n\n```yaml\n# prometheus.yml\nscrape_configs:\n - job_name: 'docker-containers'\n static_configs:\n - targets: ['app:8080', 'api:8080']\n metrics_path: '/metrics'\n```\n\n### Log Aggregation Patterns\n\n**Pattern 1: Sidecar container:**\n\n```yaml\nservices:\n app:\n image: myapp\n volumes:\n - logs:/app/logs\n\n log-shipper:\n image: fluent/fluent-bit\n volumes:\n - logs:/logs:ro\n depends_on:\n - app\n\nvolumes:\n logs:\n```\n\n**Pattern 2: Logging driver to aggregator:**\n\n```yaml\nservices:\n app:\n image: myapp\n logging:\n driver: fluentd\n options:\n fluentd-address: localhost:24224\n tag: myapp.{{.Name}}\n\n fluentd:\n image: fluent/fluentd\n ports:\n - \"24224:24224\"\n volumes:\n - ./fluent.conf:/fluentd/etc/fluent.conf\n```\n\n**Pattern 3: stdout to Loki:**\n\n```yaml\nservices:\n app:\n image: myapp\n logging:\n driver: loki\n options:\n loki-url: \"http://loki:3100/loki/api/v1/push\"\n loki-batch-size: \"400\"\n\n loki:\n image: grafana/loki\n ports:\n - \"3100:3100\"\n\n grafana:\n image: grafana/grafana\n ports:\n - \"3000:3000\"\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Logging to Files Instead of stdout\n\n```dockerfile\n# ❌ Logs go to file, invisible to docker logs\nCMD [\"./app\", \"--log-file=/var/log/app.log\"]\n\n# ✅ Logs go to stdout, captured by Docker\nCMD [\"./app\"]\n\n# If app MUST write to file, redirect to stdout\n# Create symlink: ln -sf /dev/stdout /var/log/app.log\n```\n\n### Mistake 2: No Log Rotation\n\n```yaml\n# ❌ Default json-file with no limits\nservices:\n app:\n image: myapp\n # Logs grow forever, fill disk\n\n# ✅ Always configure rotation\nservices:\n app:\n image: myapp\n logging:\n driver: json-file\n options:\n max-size: \"10m\"\n max-file: \"3\"\n```\n\n### Mistake 3: Shallow Health Checks\n\n```dockerfile\n# ❌ Only checks if process is running\nHEALTHCHECK CMD pgrep nginx\n\n# ❌ Only checks if port is open\nHEALTHCHECK CMD nc -z localhost 8080\n\n# ✅ Checks actual functionality\nHEALTHCHECK CMD curl -f http://localhost:8080/health/ready || exit 1\n```\n\n---\n\n## 🐛 Debug This: The Disappearing Logs\n\nA developer reports: \"My container logs show nothing! But I know it's writing logs!\"\n\n```bash\n$ docker logs myapp\n# (empty output)\n\n$ docker exec myapp ls /var/log/app/\n# app.log (file exists and has content)\n\n$ docker exec myapp cat /var/log/app/app.log\n# [2024-01-15] INFO Starting application...\n# [2024-01-15] INFO Processing request...\n```\n\n**Why doesn't `docker logs` show anything?**\n\n---\n\n**✅ Solution:**\n\nThe application is logging to a **file** instead of **stdout/stderr**. Docker only captures stdout/stderr streams.\n\n**Fixes:**\n\n**1. Configure application to log to stdout:**\n```bash\n# Many apps have options for this\n./app --log-target=stdout\n```\n\n**2. Redirect file to stdout in Dockerfile:**\n```dockerfile\n# Create symlink so file writes go to stdout\nRUN ln -sf /dev/stdout /var/log/app/app.log\n```\n\n**3. Use tail in entrypoint:**\n```dockerfile\n# Stream file to stdout\nCMD [\"sh\", \"-c\", \"./app & tail -f /var/log/app/app.log\"]\n```\n\n**4. Change application logging config:**\n```python\n# Python example\nimport logging\nimport sys\n\n# Log to stdout\nhandler = logging.StreamHandler(sys.stdout)\nlogger.addHandler(handler)\n```\n\n**The correct fix** is usually option 1 or 4 - configure the application properly. Options 2 and 3 are workarounds.\n\n**12-factor principle:** Applications should never manage log files. Write to stdout, let the platform handle routing and storage.\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Configure Log Rotation\n⭐ Difficulty: Easy | ⏱️ Time: 15 minutes\n\n```bash\n# Run container with default logging\ndocker run -d --name log-test alpine sh -c \"while true; do echo 'Log entry'; sleep 1; done\"\n\n# Watch logs grow\nwatch -n 5 \"du -sh /var/lib/docker/containers/$(docker inspect log-test --format '{{.Id}}')/*-json.log\"\n\n# Stop and remove\ndocker rm -f log-test\n\n# Now run with rotation\ndocker run -d --name log-test-rotated \\\n --log-opt max-size=1m \\\n --log-opt max-file=3 \\\n alpine sh -c \"while true; do echo 'Log entry $(date)'; sleep 0.1; done\"\n\n# Watch rotation happen\nwatch -n 5 \"ls -lh /var/lib/docker/containers/$(docker inspect log-test-rotated --format '{{.Id}}')/\"\n\n# Clean up\ndocker rm -f log-test-rotated\n```\n\n---\n\n### Exercise 2: Implement Health Check\n⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes\n\n```bash\nmkdir health-test && cd health-test\n\n# Create a simple server that can be \"unhealthy\"\ncat > server.py << 'EOF'\nfrom http.server import HTTPServer, BaseHTTPRequestHandler\nimport os\n\nhealthy = True\n\nclass Handler(BaseHTTPRequestHandler):\n def do_GET(self):\n global healthy\n if self.path == '/health':\n if healthy:\n self.send_response(200)\n self.end_headers()\n self.wfile.write(b'OK')\n else:\n self.send_response(503)\n self.end_headers()\n self.wfile.write(b'UNHEALTHY')\n elif self.path == '/break':\n healthy = False\n self.send_response(200)\n self.end_headers()\n self.wfile.write(b'Now unhealthy')\n else:\n self.send_response(200)\n self.end_headers()\n self.wfile.write(b'Hello')\n\nHTTPServer(('0.0.0.0', 8080), Handler).serve_forever()\nEOF\n\n# Create Dockerfile with health check\ncat > Dockerfile << 'EOF'\nFROM python:3.11-alpine\nRUN apk add --no-cache curl\nCOPY server.py /server.py\nHEALTHCHECK --interval=5s --timeout=3s --retries=3 \\\n CMD curl -f http://localhost:8080/health || exit 1\nCMD [\"python\", \"/server.py\"]\nEOF\n\n# Build and run\ndocker build -t health-test .\ndocker run -d --name health-test -p 8080:8080 health-test\n\n# Watch health status\nwatch -n 2 'docker inspect health-test --format \"{{.State.Health.Status}}\"'\n\n# In another terminal, break it\ncurl http://localhost:8080/break\n\n# Watch it become unhealthy\n# Clean up\ndocker rm -f health-test\n```\n\n---\n\n### Exercise 3: Structured Logging\n⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes\n\n```bash\nmkdir structured-logging && cd structured-logging\n\n# Create app with structured logs\ncat > app.py << 'EOF'\nimport json\nimport sys\nimport time\nimport random\nfrom datetime import datetime\n\ndef log(level, message, **extra):\n entry = {\n \"timestamp\": datetime.utcnow().isoformat() + \"Z\",\n \"level\": level,\n \"message\": message,\n **extra\n }\n print(json.dumps(entry), flush=True)\n\n# Simulate application\nwhile True:\n duration = random.uniform(0.1, 2.0)\n status = random.choice([200, 200, 200, 404, 500])\n\n log(\"INFO\", \"Request processed\",\n request_id=f\"req-{random.randint(1000,9999)}\",\n method=\"GET\",\n path=\"/api/users\",\n status=status,\n duration_ms=int(duration * 1000))\n\n if status == 500:\n log(\"ERROR\", \"Internal server error\",\n error=\"Database connection failed\")\n\n time.sleep(1)\nEOF\n\n# Run and view logs\ndocker run -d --name structured-app python:3.11 python -u /dev/stdin < app.py\ndocker logs -f structured-app\n\n# In another terminal, parse with jq\ndocker logs structured-app 2>&1 | jq 'select(.level == \"ERROR\")'\n\n# Clean up\ndocker rm -f structured-app\n```\n\n---\n\n### Exercise 4: Prometheus Metrics\n⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 30 minutes\n\nCreate a complete metrics setup:\n\n```yaml\n# docker-compose.yml\nversion: '3.8'\n\nservices:\n app:\n build: .\n ports:\n - \"8080:8080\"\n\n prometheus:\n image: prom/prometheus\n ports:\n - \"9090:9090\"\n volumes:\n - ./prometheus.yml:/etc/prometheus/prometheus.yml\n\n grafana:\n image: grafana/grafana\n ports:\n - \"3000:3000\"\n environment:\n - GF_SECURITY_ADMIN_PASSWORD=admin\n```\n\n```yaml\n# prometheus.yml\nglobal:\n scrape_interval: 15s\n\nscrape_configs:\n - job_name: 'app'\n static_configs:\n - targets: ['app:8080']\n```\n\nTasks:\n1. Create a simple app that exposes `/metrics` endpoint\n2. Include request count and duration metrics\n3. Set up Prometheus to scrape the metrics\n4. Create a simple Grafana dashboard\n\n---\n\n### Exercise 5: Complete Observability Stack\n⭐⭐⭐⭐ Difficulty: Expert | ⏱️ Time: 45 minutes\n\nBuild a complete observability setup:\n1. Application with structured JSON logging\n2. Health checks (liveness + readiness)\n3. Prometheus metrics\n4. Log aggregation with Loki\n5. Grafana dashboards for both metrics and logs\n\n```yaml\n# Requirements:\n# - App logs in JSON format to stdout\n# - /health/live endpoint (process alive)\n# - /health/ready endpoint (dependencies ok)\n# - /metrics endpoint (Prometheus format)\n# - Loki collecting logs\n# - Grafana showing both metrics and logs\n```\n\n---\n\n## 🎤 Senior-Level Interview Questions\n\n### Q1: Why should containers log to stdout instead of files?\n\n**Strong Answer:**\n\n\"This follows the twelve-factor app methodology and has practical benefits:\n\n**1. Separation of concerns:**\n- Application produces logs\n- Platform handles routing, storage, rotation\n- App doesn't need to know about log infrastructure\n\n**2. Portability:**\n- Same container works with any logging backend\n- json-file for dev, fluentd for prod, CloudWatch for AWS\n- No code changes needed\n\n**3. No log file management:**\n- No rotation logic in app\n- No disk full issues from unrotated logs\n- No volume mounts for log persistence\n\n**4. Container lifecycle:**\n- Logs available via `docker logs` immediately\n- Survive container restarts (with json-file driver)\n- Aggregatable across containers\n\n**5. Debugging:**\n- `docker logs` just works\n- No need to exec into container\n- Consistent interface across all containers\n\nThe exception is when you need file-based logging for specific tools. In that case, symlink the file to /dev/stdout: `ln -sf /dev/stdout /var/log/app.log`\"\n\n### Q2: Explain the difference between liveness and readiness health checks.\n\n**Strong Answer:**\n\n\"They serve different purposes in container orchestration:\n\n**Liveness:**\n- Question: 'Is the process alive and not deadlocked?'\n- Failure action: Restart the container\n- Should be: Fast, minimal dependencies\n- Example: Can the process respond to a simple request?\n\n```\nGET /health/live\nResponse: 200 OK (process is alive)\n```\n\n**Readiness:**\n- Question: 'Can this instance serve traffic?'\n- Failure action: Remove from load balancer, don't restart\n- Should check: Dependencies (DB, cache, upstream services)\n- Example: Are all required connections healthy?\n\n```\nGET /health/ready\nResponse: 200 if DB + cache + upstream OK\n 503 if any dependency is down\n```\n\n**Why separate them:**\n\nScenario: Database goes down temporarily.\n- Readiness fails: Container removed from LB, no traffic\n- Liveness passes: Container stays running\n- When DB recovers: Readiness passes, traffic resumes\n- No unnecessary container restarts\n\nIf we only had liveness including DB check:\n- DB down → liveness fails → container restarts\n- Restarting doesn't fix DB\n- Container keeps crash-looping\n- Worse than just waiting for DB\n\n**In Docker:**\n```dockerfile\nHEALTHCHECK --interval=30s CMD curl -f http://localhost/health/ready || exit 1\n```\n\n**In Kubernetes:**\n```yaml\nlivenessProbe:\n httpGet:\n path: /health/live\nreadinessProbe:\n httpGet:\n path: /health/ready\n```\"\n\n### Q3: How do you handle logging in a microservices architecture?\n\n**Strong Answer:**\n\n\"Centralized logging is essential for microservices. My approach:\n\n**1. Structured logging everywhere:**\n```json\n{\\\"timestamp\\\":\\\"...\\\",\\\"service\\\":\\\"user-api\\\",\\\"trace_id\\\":\\\"abc123\\\",\\\"message\\\":\\\"...\\\"}\n```\n- JSON format for machine parsing\n- Include service name in every log\n- Include trace/correlation ID\n\n**2. Distributed tracing:**\n- Propagate trace ID across service calls\n- Use OpenTelemetry or similar\n- Enables request flow visualization\n\n**3. Centralized aggregation:**\n```\nContainer → Logging driver → Collector → Storage → Query UI\n (fluentd) (Fluentd) (ES/Loki) (Kibana/Grafana)\n```\n\n**4. Log levels strategically:**\n- ERROR: Requires attention\n- WARN: Concerning but handled\n- INFO: Business events\n- DEBUG: Only in dev/troubleshooting\n\n**5. Include context:**\n```python\nlogger.info('Order processed', extra={\n 'order_id': order.id,\n 'user_id': user.id,\n 'amount': order.total,\n 'trace_id': request.trace_id\n})\n```\n\n**6. Alerting on patterns:**\n- Alert on error rate increase\n- Alert on specific error types\n- Don't alert on every error\n\nThe goal is: from any error, I can trace the entire request flow across all services.\"\n\n### Q4: How do you configure Docker logging to prevent disk space issues?\n\n**Strong Answer:**\n\n\"This is a common production issue. The solution involves multiple layers:\n\n**1. Configure log rotation (container level):**\n```yaml\nservices:\n app:\n logging:\n driver: json-file\n options:\n max-size: '10m'\n max-file: '5'\n```\nEach container: max 50MB logs (5 × 10MB)\n\n**2. Set daemon defaults (host level):**\n```json\n// /etc/docker/daemon.json\n{\n \\\"log-driver\\\": \\\"json-file\\\",\n \\\"log-opts\\\": {\n \\\"max-size\\\": \\\"10m\\\",\n \\\"max-file\\\": \\\"3\\\"\n }\n}\n```\nApplies to all containers without explicit config.\n\n**3. Monitor disk usage:**\n```bash\n# Alert when docker storage exceeds threshold\ndu -sh /var/lib/docker/containers/\n```\n\n**4. Consider alternative drivers:**\n- Production: fluentd/syslog to external storage\n- Development: json-file with rotation\n\n**5. Regular cleanup:**\n```bash\n# Prune unused data\ndocker system prune --volumes\n\n# Or scheduled\n0 0 * * * docker system prune -f\n```\n\n**6. Log level management:**\n- Production: INFO and above\n- Debug logs only when troubleshooting\n\nThe key is proactive configuration. Default json-file with no limits will eventually fill any disk.\"\n\n### Q5: How would you implement metrics collection for Docker containers?\n\n**Strong Answer:**\n\n\"I use the Prometheus pull model as the standard approach:\n\n**Application metrics:**\n```python\n# Each service exposes /metrics endpoint\nfrom prometheus_client import Counter, Histogram, generate_latest\n\nrequests_total = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])\nrequest_duration = Histogram('http_request_duration_seconds', 'Request duration')\n\n@app.route('/metrics')\ndef metrics():\n return generate_latest()\n```\n\n**Container metrics (cAdvisor):**\n```yaml\nservices:\n cadvisor:\n image: gcr.io/cadvisor/cadvisor\n volumes:\n - /:/rootfs:ro\n - /var/run:/var/run:ro\n - /sys:/sys:ro\n - /var/lib/docker/:/var/lib/docker:ro\n```\nExposes CPU, memory, network, disk metrics per container.\n\n**Prometheus configuration:**\n```yaml\nscrape_configs:\n - job_name: 'cadvisor'\n static_configs:\n - targets: ['cadvisor:8080']\n - job_name: 'applications'\n static_configs:\n - targets: ['app1:8080', 'app2:8080']\n```\n\n**Key metrics to collect:**\n- Request rate, errors, duration (RED)\n- Resource usage (CPU, memory, network)\n- Business metrics (orders, users, etc.)\n- Dependency health (DB connections, queue depth)\n\n**Visualization:**\nGrafana dashboards with:\n- Per-service panels\n- Container resource usage\n- Error rates and latencies\n- Alerting thresholds\n\n**For Kubernetes:** Use kube-state-metrics and node-exporter in addition to cAdvisor.\"\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Logging Best Practices\n\n| Practice | Implementation |\n|----------|---------------|\n| Log to stdout | App writes to console, Docker captures |\n| Structured format | JSON for machine parsing |\n| Include context | trace_id, service, request details |\n| Configure rotation | max-size and max-file options |\n| Centralize logs | Fluentd, Loki, or cloud service |\n\n### Health Check Types\n\n| Type | Checks | On Failure |\n|------|--------|------------|\n| Liveness | Process alive? | Restart container |\n| Readiness | Can serve traffic? | Remove from LB |\n| Startup | Initialized? | Delay other checks |\n\n### Observability Stack\n\n```\nApplication\n├── Logs → stdout → Logging driver → Aggregator\n├── Metrics → /metrics → Prometheus → Grafana\n└── Health → /health/* → Docker/K8s → Orchestrator\n```\n\n---\n\n## 📋 Quick Reference\n\n### Logging Commands\n\n```bash\n# View logs\ndocker logs container\ndocker logs -f container # Follow\ndocker logs --tail 100 container # Last 100\ndocker logs --since 1h container # Last hour\n\n# Configure rotation\n--log-opt max-size=10m\n--log-opt max-file=3\n```\n\n### Health Check Dockerfile\n\n```dockerfile\nHEALTHCHECK --interval=30s --timeout=10s --retries=3 \\\n --start-period=60s \\\n CMD curl -f http://localhost:8080/health || exit 1\n```\n\n### Compose Logging\n\n```yaml\nservices:\n app:\n logging:\n driver: json-file\n options:\n max-size: \"10m\"\n max-file: \"3\"\n```\n\n---\n\n## 📅 Review Schedule\n\n| Day | Task | Time |\n|-----|------|------|\n| Day 1 | Review logging driver options | 10 min |\n| Day 3 | Configure log rotation in a project | 15 min |\n| Day 7 | Implement health check | 20 min |\n| Day 14 | Set up structured logging | 25 min |\n| Day 30 | Deploy complete monitoring stack | 45 min |\n\n---\n\n## 📚 Series Navigation\n\n| Previous | Current | Next |\n|----------|---------|------|\n| [Part 10: Volumes & Storage](docker-part-10-volumes) | **Part 11: Logging & Observability** | [Part 12: Container Security](docker-part-12-security) |\n\n**Docker Compendium Series:**\n- [Part 0: How to Use This Series](docker-part-0-how-to-use-this-series)\n- [Part 1-10: Previous parts...](docker-part-1-container-internals)\n- **Part 11: Logging & Observability** ← You are here\n- [Part 12: Container Security Hardening](docker-part-12-security)\n","featuredImage","isPublished","isFeatured","subscriptionType","BASIC","publishedAt","2026-06-10T07:19:25.493525Z","createdAt","2026-06-10T07:19:25.494123Z","updatedAt","2026-06-16T21:26:56.086200Z","readingTimeMinutes",16,"estimatedReadingTime","authorId",3,"authorUsername","tomasz_main","categoryName","Devops","readingTime","seriesId","seriesOrder",11,"author",{"_33":57,"_96":59,"_97":98,"_99":100,"_101":-5,"_102":-5,"_103":-5,"_104":-5,"_105":-5},"category",{"_33":92,"_74":61,"_37":93,"_77":94,"_95":-5},"tags",[80,81,82],"series",{"_33":57,"_74":75,"_37":76,"_77":78,"_79":24},"name","Docker Compendium: From Practitioner to Expert","docker-compendium","description","Master Docker internals, optimization patterns, and production best practices. For developers who use Docker daily but want deeper understanding of what's happening under the hood. Not a beginner series - we assume you can write Dockerfiles and use docker-compose. Learn what namespaces and cgroups really do, why your builds are slow, how to debug containers that crash, and how to run Docker in production like a pro.\n","isActive",{"_33":89,"_74":90,"_37":91},{"_33":86,"_74":87,"_37":88},{"_33":83,"_74":84,"_37":85},238,"Monitoring","monitoring",237,"Logging","logging",213,"Docker","docker",7,"devops","Category for Devops","colorHex","username","firstName","Tomasz","lastName","Jarosz","profilePicture","bio","website","githubUrl","linkedinUrl",69,"Monitoring & Troubleshooting Production","docker-part-19-monitoring","Container metrics, resource analysis, post-mortem","## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| **Difficulty** | 🟠 Advanced |\n| **Prerequisites** | Part 11 (Logging), Part 13 (Debugging) |\n| **Stack** | Prometheus, Grafana, Loki, cAdvisor |\n| **Time Investment** | 28 minutes read + 90 minutes practice |\n| **Payoff** | Find issues before users report them |\n\n---\n\n## 🎯 What You'll Learn\n\nAfter this article, you'll be able to:\n\n1. **Set up metrics collection** with Prometheus and cAdvisor\n2. **Build dashboards** in Grafana for container monitoring\n3. **Implement centralized logging** with Loki\n4. **Create alerts** for production issues\n5. **Systematically troubleshoot** production problems\n\n---\n\n## 🔥 Production Story: The Silent Memory Leak\n\n**The Setup**: E-commerce site, holiday season. Everything running on Docker Compose. No monitoring \"because we're small.\"\n\n**Day 1**: Site runs fine.\n**Day 3**: Occasional slow responses.\n**Day 5**: 503 errors during peak hours.\n**Day 7**: Complete outage at 2 AM. Black Friday.\n\n**The Investigation** (too late):\n```bash\n$ dmesg | tail\n[timestamp] Out of memory: Kill process 1234 (java) score 900\n[timestamp] Out of memory: Kill process 1235 (node) score 850\n\n$ docker stats --no-stream\nNAME MEM USAGE / LIMIT\napi 1.9GB / 2GB # 95% memory!\nworker 1.8GB / 2GB # 90% memory!\n```\n\n**Root Cause**: Memory leak in API service. Without monitoring, nobody noticed the gradual increase until it was too late.\n\n**What Monitoring Would Have Shown**:\n```\nDay 1: api memory = 400MB ✓\nDay 3: api memory = 1.2GB ⚠️ Alert triggered\nDay 5: api memory = 1.6GB (restart, investigate)\n```\n\n**The Fix**: Prometheus + Grafana + alerts. Now they catch issues before users do.\n\n---\n\n## 🧠 Mental Model: Observability Stack\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ OBSERVABILITY PYRAMID │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ DASHBOARDS │ │\n│ │ (Grafana - Visualization) │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ ↑ │\n│ ┌──────────────┬────────┴────────┬──────────────┐ │\n│ │ METRICS │ LOGS │ TRACES │ │\n│ │ Prometheus │ Loki │ Jaeger │ │\n│ │ │ │ (optional) │ │\n│ └──────────────┴─────────────────┴──────────────┘ │\n│ ↑ │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ COLLECTORS │ │\n│ │ cAdvisor node_exporter Promtail App metrics │ │\n│ │ (containers) (host) (logs) (custom) │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ ↑ │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ DATA SOURCES │ │\n│ │ Docker Daemon Host OS Application Network │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n\nData Flow:\n Containers → cAdvisor → Prometheus → Grafana → Alerts\n ↓\n Logs → Promtail → Loki → Grafana\n```\n\n---\n\n## 🔬 Deep Dive\n\n### 1. Metrics Collection with cAdvisor and Prometheus\n\n**Docker Compose Monitoring Stack**:\n\n```yaml\n# docker-compose.monitoring.yml\n\nservices:\n # ═══════════════════════════════════════════════════════════════\n # PROMETHEUS - Metrics database\n # ═══════════════════════════════════════════════════════════════\n prometheus:\n image: prom/prometheus:v2.47.0\n command:\n - '--config.file=/etc/prometheus/prometheus.yml'\n - '--storage.tsdb.path=/prometheus'\n - '--storage.tsdb.retention.time=15d'\n - '--web.enable-lifecycle'\n volumes:\n - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro\n - ./prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro\n - prometheus_data:/prometheus\n ports:\n - \"9090:9090\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # GRAFANA - Visualization\n # ═══════════════════════════════════════════════════════════════\n grafana:\n image: grafana/grafana:10.1.0\n environment:\n - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}\n - GF_USERS_ALLOW_SIGN_UP=false\n volumes:\n - grafana_data:/var/lib/grafana\n - ./grafana/provisioning:/etc/grafana/provisioning:ro\n - ./grafana/dashboards:/var/lib/grafana/dashboards:ro\n ports:\n - \"3000:3000\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # cADVISOR - Container metrics\n # ═══════════════════════════════════════════════════════════════\n cadvisor:\n image: gcr.io/cadvisor/cadvisor:v0.47.2\n privileged: true\n volumes:\n - /:/rootfs:ro\n - /var/run:/var/run:ro\n - /sys:/sys:ro\n - /var/lib/docker/:/var/lib/docker:ro\n - /dev/disk/:/dev/disk:ro\n ports:\n - \"8080:8080\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # NODE EXPORTER - Host metrics\n # ═══════════════════════════════════════════════════════════════\n node-exporter:\n image: prom/node-exporter:v1.6.1\n command:\n - '--path.rootfs=/host'\n volumes:\n - /:/host:ro,rslave\n ports:\n - \"9100:9100\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # ALERTMANAGER - Alert routing\n # ═══════════════════════════════════════════════════════════════\n alertmanager:\n image: prom/alertmanager:v0.26.0\n volumes:\n - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro\n ports:\n - \"9093:9093\"\n networks:\n - monitoring\n restart: unless-stopped\n\nnetworks:\n monitoring:\n driver: bridge\n\nvolumes:\n prometheus_data:\n grafana_data:\n```\n\n**Prometheus Configuration**:\n\n```yaml\n# prometheus/prometheus.yml\n\nglobal:\n scrape_interval: 15s\n evaluation_interval: 15s\n\nalerting:\n alertmanagers:\n - static_configs:\n - targets: ['alertmanager:9093']\n\nrule_files:\n - /etc/prometheus/alerts.yml\n\nscrape_configs:\n # Prometheus itself\n - job_name: 'prometheus'\n static_configs:\n - targets: ['localhost:9090']\n\n # Container metrics from cAdvisor\n - job_name: 'cadvisor'\n static_configs:\n - targets: ['cadvisor:8080']\n\n # Host metrics from node-exporter\n - job_name: 'node'\n static_configs:\n - targets: ['node-exporter:9100']\n\n # Docker daemon metrics\n - job_name: 'docker'\n static_configs:\n - targets: ['host.docker.internal:9323']\n\n # Application metrics (if exposed)\n - job_name: 'app'\n static_configs:\n - targets: ['api:8080']\n metrics_path: /metrics\n```\n\n---\n\n### 2. Key Metrics to Monitor\n\n**Container Metrics** (from cAdvisor):\n\n```promql\n# CPU Usage (percentage of limit)\nsum(rate(container_cpu_usage_seconds_total{name=~\".+\"}[5m])) by (name) * 100\n\n# Memory Usage (bytes)\ncontainer_memory_usage_bytes{name=~\".+\"}\n\n# Memory Usage Percentage\ncontainer_memory_usage_bytes{name=~\".+\"} / container_spec_memory_limit_bytes{name=~\".+\"} * 100\n\n# Network I/O\nrate(container_network_receive_bytes_total{name=~\".+\"}[5m])\nrate(container_network_transmit_bytes_total{name=~\".+\"}[5m])\n\n# Disk I/O\nrate(container_fs_reads_bytes_total{name=~\".+\"}[5m])\nrate(container_fs_writes_bytes_total{name=~\".+\"}[5m])\n\n# Container restarts\nincrease(container_start_time_seconds{name=~\".+\"}[1h])\n```\n\n**Host Metrics** (from node-exporter):\n\n```promql\n# CPU Usage\n100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)\n\n# Memory Usage\n(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100\n\n# Disk Usage\n100 - ((node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"}) * 100)\n\n# Network\nrate(node_network_receive_bytes_total[5m])\nrate(node_network_transmit_bytes_total[5m])\n```\n\n---\n\n### 3. Alert Rules\n\n```yaml\n# prometheus/alerts.yml\n\ngroups:\n - name: container_alerts\n rules:\n # Container down\n - alert: ContainerDown\n expr: absent(container_last_seen{name=~\".+\"}) == 1\n for: 1m\n labels:\n severity: critical\n annotations:\n summary: \"Container {{ $labels.name }} is down\"\n description: \"Container has been down for more than 1 minute\"\n\n # High memory usage\n - alert: ContainerHighMemory\n expr: |\n (container_memory_usage_bytes{name=~\".+\"} /\n container_spec_memory_limit_bytes{name=~\".+\"}) > 0.85\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"Container {{ $labels.name }} memory usage > 85%\"\n description: \"Memory usage is {{ $value | printf \\\"%.1f\\\" }}%\"\n\n # Memory near limit (critical)\n - alert: ContainerMemoryCritical\n expr: |\n (container_memory_usage_bytes{name=~\".+\"} /\n container_spec_memory_limit_bytes{name=~\".+\"}) > 0.95\n for: 1m\n labels:\n severity: critical\n annotations:\n summary: \"Container {{ $labels.name }} memory critical\"\n description: \"Memory at {{ $value | printf \\\"%.1f\\\" }}%, OOM likely\"\n\n # High CPU usage\n - alert: ContainerHighCPU\n expr: |\n sum(rate(container_cpu_usage_seconds_total{name=~\".+\"}[5m])) by (name) > 0.8\n for: 10m\n labels:\n severity: warning\n annotations:\n summary: \"Container {{ $labels.name }} high CPU usage\"\n description: \"CPU usage is {{ $value | printf \\\"%.1f\\\" }}%\"\n\n # Container restarts\n - alert: ContainerRestarting\n expr: increase(container_start_time_seconds{name=~\".+\"}[1h]) > 3\n labels:\n severity: warning\n annotations:\n summary: \"Container {{ $labels.name }} restarting frequently\"\n description: \"{{ $value }} restarts in the last hour\"\n\n - name: host_alerts\n rules:\n # High host memory\n - alert: HostHighMemory\n expr: |\n (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) /\n node_memory_MemTotal_bytes > 0.9\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"Host memory usage > 90%\"\n\n # Disk space low\n - alert: HostDiskSpaceLow\n expr: |\n (node_filesystem_avail_bytes{mountpoint=\"/\"} /\n node_filesystem_size_bytes{mountpoint=\"/\"}) < 0.1\n for: 5m\n labels:\n severity: critical\n annotations:\n summary: \"Disk space < 10% remaining\"\n```\n\n**Alertmanager Configuration**:\n\n```yaml\n# alertmanager/alertmanager.yml\n\nglobal:\n resolve_timeout: 5m\n\nroute:\n group_by: ['alertname']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 1h\n receiver: 'slack-notifications'\n\n routes:\n - match:\n severity: critical\n receiver: 'pagerduty-critical'\n - match:\n severity: warning\n receiver: 'slack-notifications'\n\nreceivers:\n - name: 'slack-notifications'\n slack_configs:\n - api_url: '${SLACK_WEBHOOK_URL}'\n channel: '#alerts'\n title: '{{ .Status | toUpper }}: {{ .CommonAnnotations.summary }}'\n text: '{{ .CommonAnnotations.description }}'\n\n - name: 'pagerduty-critical'\n pagerduty_configs:\n - service_key: '${PAGERDUTY_KEY}'\n severity: critical\n```\n\n---\n\n### 4. Centralized Logging with Loki\n\n**Add Loki to Monitoring Stack**:\n\n```yaml\n# Add to docker-compose.monitoring.yml\n\nservices:\n loki:\n image: grafana/loki:2.9.0\n command: -config.file=/etc/loki/local-config.yaml\n volumes:\n - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro\n - loki_data:/loki\n ports:\n - \"3100:3100\"\n networks:\n - monitoring\n restart: unless-stopped\n\n promtail:\n image: grafana/promtail:2.9.0\n volumes:\n - ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro\n - /var/lib/docker/containers:/var/lib/docker/containers:ro\n - /var/run/docker.sock:/var/run/docker.sock\n command: -config.file=/etc/promtail/config.yml\n networks:\n - monitoring\n restart: unless-stopped\n\nvolumes:\n loki_data:\n```\n\n**Promtail Configuration**:\n\n```yaml\n# promtail/promtail-config.yml\n\nserver:\n http_listen_port: 9080\n grpc_listen_port: 0\n\npositions:\n filename: /tmp/positions.yaml\n\nclients:\n - url: http://loki:3100/loki/api/v1/push\n\nscrape_configs:\n - job_name: docker\n docker_sd_configs:\n - host: unix:///var/run/docker.sock\n refresh_interval: 5s\n relabel_configs:\n # Use container name as label\n - source_labels: ['__meta_docker_container_name']\n regex: '/(.*)'\n target_label: 'container'\n # Add compose service label\n - source_labels: ['__meta_docker_container_label_com_docker_compose_service']\n target_label: 'service'\n # Add compose project label\n - source_labels: ['__meta_docker_container_label_com_docker_compose_project']\n target_label: 'project'\n```\n\n**Loki Configuration**:\n\n```yaml\n# loki/loki-config.yml\n\nauth_enabled: false\n\nserver:\n http_listen_port: 3100\n\ningester:\n lifecycler:\n ring:\n kvstore:\n store: inmemory\n replication_factor: 1\n chunk_idle_period: 5m\n chunk_retain_period: 30s\n\nschema_config:\n configs:\n - from: 2020-01-01\n store: boltdb-shipper\n object_store: filesystem\n schema: v11\n index:\n prefix: index_\n period: 24h\n\nstorage_config:\n boltdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/cache\n shared_store: filesystem\n filesystem:\n directory: /loki/chunks\n\nlimits_config:\n enforce_metric_name: false\n reject_old_samples: true\n reject_old_samples_max_age: 168h\n\nchunk_store_config:\n max_look_back_period: 0s\n\ntable_manager:\n retention_deletes_enabled: true\n retention_period: 168h # 7 days\n```\n\n**LogQL Queries in Grafana**:\n\n```logql\n# All logs from a container\n{container=\"api\"}\n\n# Error logs only\n{container=\"api\"} |= \"error\"\n\n# JSON log parsing\n{container=\"api\"} | json | level=\"error\"\n\n# Count errors per minute\ncount_over_time({container=\"api\"} |= \"error\" [1m])\n\n# Top error messages\n{container=\"api\"} | json | level=\"error\"\n | line_format \"{{ .message }}\"\n | topk(10, count_over_time({container=\"api\"} [1h]) by (message))\n\n# Response time from logs (if logged)\n{container=\"api\"} | json | unwrap duration [5m]\n | quantile_over_time(0.95, 5m)\n```\n\n---\n\n### 5. Grafana Dashboard for Docker\n\n**Dashboard JSON** (simplified):\n\n```json\n{\n \"title\": \"Docker Container Monitoring\",\n \"panels\": [\n {\n \"title\": \"Container CPU Usage\",\n \"type\": \"timeseries\",\n \"targets\": [\n {\n \"expr\": \"sum(rate(container_cpu_usage_seconds_total{name=~\\\".+\\\"}[5m])) by (name) * 100\",\n \"legendFormat\": \"{{name}}\"\n }\n ]\n },\n {\n \"title\": \"Container Memory Usage\",\n \"type\": \"timeseries\",\n \"targets\": [\n {\n \"expr\": \"container_memory_usage_bytes{name=~\\\".+\\\"} / 1024 / 1024\",\n \"legendFormat\": \"{{name}}\"\n }\n ]\n },\n {\n \"title\": \"Container Memory % of Limit\",\n \"type\": \"gauge\",\n \"targets\": [\n {\n \"expr\": \"(container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100\",\n \"legendFormat\": \"{{name}}\"\n }\n ],\n \"thresholds\": [\n { \"value\": 0, \"color\": \"green\" },\n { \"value\": 70, \"color\": \"yellow\" },\n { \"value\": 85, \"color\": \"red\" }\n ]\n },\n {\n \"title\": \"Container Network I/O\",\n \"type\": \"timeseries\",\n \"targets\": [\n {\n \"expr\": \"rate(container_network_receive_bytes_total{name=~\\\".+\\\"}[5m])\",\n \"legendFormat\": \"{{name}} RX\"\n },\n {\n \"expr\": \"rate(container_network_transmit_bytes_total{name=~\\\".+\\\"}[5m])\",\n \"legendFormat\": \"{{name}} TX\"\n }\n ]\n },\n {\n \"title\": \"Container Restarts (1h)\",\n \"type\": \"stat\",\n \"targets\": [\n {\n \"expr\": \"increase(container_start_time_seconds{name=~\\\".+\\\"}[1h])\",\n \"legendFormat\": \"{{name}}\"\n }\n ]\n }\n ]\n}\n```\n\n**Grafana Provisioning**:\n\n```yaml\n# grafana/provisioning/datasources/datasources.yml\n\napiVersion: 1\n\ndatasources:\n - name: Prometheus\n type: prometheus\n access: proxy\n url: http://prometheus:9090\n isDefault: true\n\n - name: Loki\n type: loki\n access: proxy\n url: http://loki:3100\n```\n\n---\n\n### 6. Troubleshooting Runbook\n\n**Systematic Troubleshooting Process**:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ TROUBLESHOOTING FLOWCHART │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ SYMPTOM DETECTED │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Check Dashboard │ → Memory, CPU, Network, Errors │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Check Logs │ → Recent errors, patterns │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Check Resources │ → docker stats, df -h, free │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Deep Dive │ → exec, inspect, strace │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Take Action │ → Restart, Scale, Rollback │\n│ └─────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**Quick Diagnosis Script**:\n\n```bash\n#!/bin/bash\n# diagnose.sh - Quick production diagnosis\n\necho \"═══════════════════════════════════════════════════════════════\"\necho \"DOCKER PRODUCTION DIAGNOSIS\"\necho \"Time: $(date)\"\necho \"═══════════════════════════════════════════════════════════════\"\n\necho -e \"\\n### Container Status ###\"\ndocker compose ps\n\necho -e \"\\n### Container Resources ###\"\ndocker stats --no-stream --format \"table {{.Name}}\\t{{.CPUPerc}}\\t{{.MemUsage}}\\t{{.MemPerc}}\"\n\necho -e \"\\n### Unhealthy Containers ###\"\ndocker ps --filter health=unhealthy --format \"{{.Names}}: {{.Status}}\"\n\necho -e \"\\n### Recent Restarts ###\"\ndocker ps --filter \"status=restarting\" --format \"{{.Names}}: {{.Status}}\"\n\necho -e \"\\n### Disk Usage ###\"\ndf -h / /var/lib/docker\n\necho -e \"\\n### Docker Disk Usage ###\"\ndocker system df\n\necho -e \"\\n### Recent Docker Events ###\"\ndocker events --since \"10m\" --until \"0s\" 2>/dev/null | tail -20\n\necho -e \"\\n### Recent Error Logs (all containers) ###\"\ndocker compose logs --since \"10m\" 2>&1 | grep -i -E \"(error|exception|fatal)\" | tail -20\n\necho -e \"\\n### Host Memory ###\"\nfree -h\n\necho -e \"\\n### Host Load ###\"\nuptime\n```\n\n---\n\n### 7. Common Issues and Solutions\n\n**Issue: Container OOM Killed**\n\n```bash\n# Detect\ndocker inspect container --format='{{.State.OOMKilled}}'\ndmesg | grep -i \"killed process\"\n\n# Diagnose\ndocker stats container --no-stream\ndocker exec container cat /sys/fs/cgroup/memory.current\ndocker exec container cat /sys/fs/cgroup/memory.max\n\n# Fix options\n1. Increase memory limit in compose\n2. Fix memory leak in application\n3. Optimize application memory usage (JVM: -XX:MaxRAMPercentage)\n```\n\n**Issue: Container Restarting Loop**\n\n```bash\n# Detect\ndocker ps # Shows \"Restarting (1) 5 seconds ago\"\ndocker inspect container --format='{{.RestartCount}}'\n\n# Diagnose\ndocker logs container --tail 100\ndocker inspect container --format='{{.State.ExitCode}}'\n# Common exit codes:\n# 137 = OOM or SIGKILL\n# 1 = Application error\n# 126 = Permission denied\n# 127 = Command not found\n\n# Fix\ndocker compose stop container\ndocker logs container # Check logs\n# Fix the issue, then:\ndocker compose up -d container\n```\n\n**Issue: High CPU Usage**\n\n```bash\n# Detect\ndocker stats --no-stream | sort -k3 -h -r | head -5\n\n# Diagnose\n# Get PID in container\ndocker exec container top -b -n 1\n\n# Profile\ndocker exec container sh -c \"\n while true; do\n date >> /tmp/profile.txt\n cat /proc/loadavg >> /tmp/profile.txt\n ps aux >> /tmp/profile.txt\n sleep 5\n done\n\" &\n\n# Fix options\n1. Scale horizontally (more containers)\n2. Add CPU limits to prevent hogging\n3. Profile application code\n```\n\n**Issue: Network Connectivity**\n\n```bash\n# Detect\ndocker exec container curl -v http://other-service:8080\n\n# Diagnose\n# Check DNS\ndocker exec container nslookup other-service\n\n# Check network\ndocker network ls\ndocker network inspect network_name\n\n# Check from another container\ndocker run --rm --network=mynetwork nicolaka/netshoot \\\n curl -v http://other-service:8080\n\n# Fix options\n1. Ensure containers on same network\n2. Check service name spelling\n3. Check if target service is healthy\n```\n\n---\n\n### 8. Production Monitoring Checklist\n\n```yaml\n# monitoring-checklist.yml\n\n# Essential Metrics to Monitor\nmetrics:\n container:\n - memory_usage_percent # Alert at 85%\n - cpu_usage_percent # Alert at 80%\n - restart_count # Alert at > 3/hour\n - health_status # Alert on unhealthy\n\n host:\n - disk_usage_percent # Alert at 90%\n - memory_usage_percent # Alert at 90%\n - cpu_usage_percent # Alert at 90%\n - network_errors # Alert on increase\n\n application:\n - request_latency_p99 # Alert at > 1s\n - error_rate # Alert at > 1%\n - request_rate # Alert on sudden drop\n\n# Essential Logs to Watch\nlogs:\n - level: error # All errors\n - level: warn # Warnings\n - message: \"OOM\" # Memory issues\n - message: \"connection refused\" # Network issues\n - message: \"timeout\" # Timeout issues\n\n# Dashboard Panels\ndashboards:\n overview:\n - Container status grid\n - CPU usage by container\n - Memory usage by container\n - Network I/O\n - Recent errors\n\n per_service:\n - Request rate\n - Error rate\n - Latency percentiles\n - Resource usage\n - Logs panel\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: No Alerts Until Outage\n\n```yaml\n# ❌ Wrong: Just dashboards, no alerts\n# Nobody watches dashboards at 3 AM\n\n# ✅ Right: Alerts for key metrics\n- alert: ContainerHighMemory\n expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85\n for: 5m\n```\n\n### Mistake 2: Alert Fatigue\n\n```yaml\n# ❌ Wrong: Alert on everything\n- alert: MinorWarning\n expr: cpu > 50% # Too sensitive\n for: 1m # Too quick\n\n# ✅ Right: Meaningful thresholds\n- alert: HighCPU\n expr: cpu > 85% # Actually concerning\n for: 10m # Sustained issue\n```\n\n### Mistake 3: No Log Retention Policy\n\n```yaml\n# ❌ Wrong: Keep logs forever\n# Disk fills up, system crashes\n\n# ✅ Right: Set retention limits\nservices:\n app:\n logging:\n options:\n max-size: \"10m\"\n max-file: \"3\"\n\n# And in Loki\nlimits_config:\n retention_period: 168h # 7 days\n```\n\n### Mistake 4: Missing Context in Logs\n\n```javascript\n// ❌ Wrong: Generic errors\nconsole.error(\"Error processing request\");\n\n// ✅ Right: Context-rich logs\nlogger.error({\n message: \"Error processing request\",\n requestId: req.id,\n userId: req.user?.id,\n path: req.path,\n error: err.message,\n stack: err.stack\n});\n```\n\n### Mistake 5: No Baseline\n\n```yaml\n# ❌ Wrong: Alert on absolute value\n- alert: HighMemory\n expr: container_memory_usage_bytes > 1000000000 # 1GB\n # What if the limit is 2GB? Or 500MB?\n\n# ✅ Right: Alert on percentage of limit\n- alert: HighMemory\n expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85\n```\n\n---\n\n## 🐛 Debug This\n\nProduction alert: \"Container api memory > 90%\". Dashboard shows memory climbing steadily for the past 3 days.\n\n```\nDay 1: 40% → Day 2: 60% → Day 3: 80% → Now: 92%\n```\n\nNo code changes were deployed. What's happening and how do you fix it?\n\n

Click to reveal analysis

\n\n**Diagnosis**:\nThis is a classic memory leak pattern - steady growth without code changes means:\n1. Application-level memory leak\n2. Connection leak (database, redis, etc.)\n3. Cache without eviction\n4. Event listeners not cleaned up\n\n**Investigation steps**:\n\n```bash\n# 1. Get memory breakdown\ndocker exec api node --expose-gc -e \"\n const v8 = require('v8');\n console.log(v8.getHeapStatistics());\n\"\n\n# 2. Check connection counts\ndocker exec api netstat -an | grep ESTABLISHED | wc -l\n\n# 3. Check database connections\ndocker exec api sh -c \"\n curl http://db:5432/pools # If using pgBouncer\n\"\n\n# 4. Generate heap snapshot (Node.js)\ndocker exec api kill -USR2 1 # If app supports it\n\n# 5. Check for patterns\ndocker compose logs api --since \"72h\" | grep -i \"leak\\|memory\\|heap\"\n```\n\n**Common fixes**:\n\n1. **Restart immediately** to restore service:\n ```bash\n docker compose restart api\n ```\n\n2. **Investigate root cause**:\n - Add memory profiling to application\n - Check for unclosed database connections\n - Look for growing arrays/objects\n - Check event listener cleanup\n\n3. **Add protection**:\n ```yaml\n # Compose: auto-restart on high memory\n healthcheck:\n test: |\n MEMORY=$(cat /sys/fs/cgroup/memory.current)\n LIMIT=$(cat /sys/fs/cgroup/memory.max)\n if [ $((MEMORY * 100 / LIMIT)) -gt 90 ]; then exit 1; fi\n curl -f http://localhost:8080/health\n ```\n\n4. **Long-term fix**: Profile application, find and fix the leak.\n

\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Basic Monitoring Stack\nSet up Prometheus + Grafana + cAdvisor for a simple web application. Create a dashboard showing:\n- Container CPU usage\n- Container memory usage\n- Network I/O\n\n### Exercise 2: Alerting\nAdd alertmanager and configure alerts for:\n- Container memory > 85%\n- Container restarting > 3 times/hour\n- Host disk space < 10%\n\n### Exercise 3: Centralized Logging\nAdd Loki and Promtail. Create LogQL queries for:\n- All errors from a specific service\n- Request latency > 1s\n- Count of errors per minute\n\n### Exercise 4: Custom Metrics\nAdd application metrics (request count, latency histogram) and:\n- Expose them on /metrics endpoint\n- Scrape with Prometheus\n- Create dashboard panel\n\n### Exercise 5: Runbook\nCreate a troubleshooting runbook for:\n- High memory usage\n- Service unreachable\n- High error rate\nInclude detection, diagnosis, and remediation steps.\n\n---\n\n## 🎤 Interview Questions\n\n### Q1: How do you monitor Docker containers in production?\n\n**Answer**: A complete monitoring stack includes:\n\n1. **Metrics Collection**:\n - cAdvisor for container metrics (CPU, memory, network, disk)\n - node-exporter for host metrics\n - Application metrics exposed on /metrics\n\n2. **Storage and Querying**:\n - Prometheus for metrics\n - Loki for logs\n - Retention policies for both\n\n3. **Visualization**:\n - Grafana dashboards\n - Key panels: resource usage, error rates, latency\n\n4. **Alerting**:\n - Prometheus alerting rules\n - Alertmanager for routing\n - Slack/PagerDuty integration\n\nKey metrics to watch:\n- Memory usage % of limit (alert at 85%)\n- Container restart count\n- Error rate and latency\n- Disk space\n\n---\n\n### Q2: How do you troubleshoot a container that's using high CPU?\n\n**Answer**: Systematic approach:\n\n1. **Identify** the container:\n ```bash\n docker stats --no-stream | sort -k3 -h -r\n ```\n\n2. **Check** what process is consuming CPU:\n ```bash\n docker exec container top -b -n 1\n docker exec container ps aux --sort=-%cpu\n ```\n\n3. **Analyze** the application:\n ```bash\n # For Node.js\n docker exec container node --prof-process\n # For Java\n docker exec container jstack PID\n ```\n\n4. **Check** for patterns:\n - Spike vs sustained high CPU\n - Correlation with traffic\n - Recent changes\n\n5. **Take action**:\n - Scale horizontally if legitimate load\n - Add CPU limits to prevent hogging\n - Fix code if inefficiency found\n - Rollback if recent deployment caused it\n\n---\n\n### Q3: What's the difference between container memory usage and the memory limit?\n\n**Answer**:\n\n**Memory Usage** (`container_memory_usage_bytes`):\n- Current memory used by the container\n- Includes RSS, cache, swap\n\n**Memory Limit** (`container_spec_memory_limit_bytes`):\n- Maximum allowed by cgroup\n- Set via `--memory` or compose `deploy.resources.limits.memory`\n\n**Why this matters**:\n\n```promql\n# Percentage is more meaningful than absolute value\ncontainer_memory_usage_bytes / container_spec_memory_limit_bytes * 100\n```\n\nWhen usage approaches limit:\n- 70-85%: Warning zone\n- 85-95%: Critical, action needed\n- >95%: OOM kill imminent\n\n**Best practice**: Alert on percentage, not absolute value. A container using 500MB might be fine (if limit is 2GB) or critical (if limit is 512MB).\n\n---\n\n### Q4: How do you implement log aggregation for Docker containers?\n\n**Answer**: Using Loki stack:\n\n1. **Deploy Loki** (log database):\n ```yaml\n services:\n loki:\n image: grafana/loki:2.9.0\n ports:\n - \"3100:3100\"\n ```\n\n2. **Deploy Promtail** (log collector):\n ```yaml\n promtail:\n image: grafana/promtail:2.9.0\n volumes:\n - /var/lib/docker/containers:/var/lib/docker/containers:ro\n - /var/run/docker.sock:/var/run/docker.sock\n ```\n\n3. **Configure discovery**:\n ```yaml\n scrape_configs:\n - job_name: docker\n docker_sd_configs:\n - host: unix:///var/run/docker.sock\n ```\n\n4. **Query in Grafana**:\n ```logql\n {container=\"api\"} |= \"error\"\n {service=\"api\"} | json | level=\"error\"\n ```\n\n**Key considerations**:\n- Set retention limits (default 7 days)\n- Index only necessary labels\n- Use JSON logging for structured queries\n\n---\n\n### Q5: How do you prevent alert fatigue while still catching real issues?\n\n**Answer**: Strategic alert configuration:\n\n1. **Meaningful thresholds**:\n ```yaml\n # Bad: Alerts on minor fluctuations\n expr: cpu > 50%\n\n # Good: Alerts on sustained issues\n expr: cpu > 85%\n for: 10m\n ```\n\n2. **Appropriate severity levels**:\n - Critical: Pages on-call immediately\n - Warning: Slack notification, review next day\n - Info: Dashboard only\n\n3. **Alert on symptoms, not causes**:\n ```yaml\n # Better: User-facing impact\n - alert: HighErrorRate\n expr: rate(http_errors[5m]) > 0.01\n\n # Worse: Internal metric\n - alert: HighThreadCount\n expr: threads > 100\n ```\n\n4. **Group related alerts**:\n ```yaml\n route:\n group_by: ['service', 'severity']\n group_wait: 30s # Wait to group related alerts\n ```\n\n5. **Regular review**: If an alert never fires, consider removing. If it fires daily and is ignored, fix threshold.\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Monitoring Stack\n- **Metrics**: Prometheus + cAdvisor\n- **Logs**: Loki + Promtail\n- **Visualization**: Grafana\n- **Alerts**: Alertmanager\n\n### Key Metrics\n| Metric | Warning | Critical |\n|--------|---------|----------|\n| Memory % | 85% | 95% |\n| CPU % | 80% | 90% |\n| Disk % | 80% | 90% |\n| Restart count | 3/hour | 5/hour |\n\n### Troubleshooting Flow\n1. Check dashboards\n2. Check logs\n3. Check resources\n4. Deep dive (exec, inspect)\n5. Take action (restart, scale, rollback)\n\n---\n\n## 📋 Quick Reference\n\n```bash\n# Quick diagnosis\ndocker stats --no-stream\ndocker ps --filter health=unhealthy\ndocker compose logs --since 10m | grep -i error\n\n# Memory check\ndocker inspect $C --format='{{.State.OOMKilled}}'\n\n# CPU check\ndocker exec $C top -b -n 1\n\n# Network check\ndocker exec $C nslookup service-name\n```\n\n```promql\n# Key PromQL queries\ncontainer_memory_usage_bytes / container_spec_memory_limit_bytes * 100\nrate(container_cpu_usage_seconds_total[5m]) * 100\nincrease(container_start_time_seconds[1h])\n```\n\n---\n\n## 📅 Review Schedule\n\n- **Day 1**: Set up Prometheus + Grafana + cAdvisor\n- **Day 3**: Create basic dashboard\n- **Day 7**: Add alerting rules\n- **Day 14**: Add centralized logging\n- **Day 30**: Build complete runbook\n\n---\n\n## 📚 Series Navigation\n\n- **Previous**: [Part 18 - Production Deployment Patterns](/articles/docker-part-18-deployment)\n- **Next**: [Part 20 - Docker Cheatsheet & Decision Guide](/articles/docker-part-20-cheatsheet)\n- **Index**: [Docker Compendium Series](/series/docker-compendium)","2026-06-10T07:19:25.577113Z","2026-06-10T07:19:25.577756Z",19,{"_33":57,"_96":59,"_97":98,"_99":100,"_101":-5,"_102":-5,"_103":-5,"_104":-5,"_105":-5},{"_33":92,"_74":61,"_37":93,"_77":94,"_95":-5},[118,119,120],{"_33":57,"_74":75,"_37":76,"_77":78,"_79":24},{"_33":89,"_74":90,"_37":91},{"_33":83,"_74":84,"_37":85},{"_33":121,"_74":122,"_37":123},216,"Production","production"]