[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6},"actionData","errors","routes/blog-tag",{"_7":8,"_9":10},"tag",{"_34":83,"_74":84,"_38":85},"articles",{"_11":12,"_13":14,"_15":16,"_17":18,"_19":20,"_21":22,"_23":24,"_25":24,"_26":16,"_27":28},"content",[31,32,33],"pageable",{"_29":22,"_30":20},"totalElements",3,"totalPages",1,"size",10,"number",0,"first",true,"last","numberOfElements","empty",false,"page_number","page_size",{"_34":124,"_36":125,"_38":126,"_40":127,"_11":128,"_43":-5,"_44":24,"_45":28,"_46":47,"_48":129,"_50":130,"_52":53,"_54":65,"_56":65,"_57":16,"_58":59,"_60":61,"_62":65,"_63":16,"_64":131,"_66":132,"_68":133,"_70":134,"_72":135},{"_34":106,"_36":107,"_38":108,"_40":109,"_11":110,"_43":-5,"_44":24,"_45":28,"_46":47,"_48":111,"_50":112,"_52":53,"_54":65,"_56":65,"_57":16,"_58":59,"_60":61,"_62":65,"_63":16,"_64":113,"_66":114,"_68":115,"_70":116,"_72":117},{"_34":35,"_36":37,"_38":39,"_40":41,"_11":42,"_43":-5,"_44":24,"_45":28,"_46":47,"_48":49,"_50":51,"_52":53,"_54":55,"_56":55,"_57":16,"_58":59,"_60":61,"_62":55,"_63":16,"_64":65,"_66":67,"_68":69,"_70":71,"_72":73},"id",66,"title","Development vs Production Compose","slug","docker-part-16-dev-vs-prod","excerpt","Override files, env-specific configs, secrets","## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| **Difficulty** | 🟡 Intermediate |\n| **Prerequisites** | Part 14 (Compose Deep Dive), Part 15 (Dependencies) |\n| **Key Patterns** | Override files, profiles, multi-stage builds |\n| **Time Investment** | 25 minutes read + 60 minutes practice |\n| **Payoff** | Same codebase works perfectly in dev and prod |\n\n---\n\n## 🎯 What You'll Learn\n\nAfter this article, you'll be able to:\n\n1. **Structure compose files** for multiple environments\n2. **Configure development** with hot reload, debugging, and local tools\n3. **Configure production** with security, performance, and reliability\n4. **Handle secrets** differently across environments\n5. **Implement environment parity** while maintaining dev productivity\n\n---\n\n## 🔥 Production Story: The \"Works on My Machine\" Disaster\n\n**The Setup**: A team used one compose file for everything. Development worked great. Production... not so much.\n\n**The Problem Compose File**:\n```yaml\nservices:\n api:\n build: .\n volumes:\n - .:/app # Great for dev hot reload\n - /app/node_modules\n ports:\n - \"3000:3000\"\n - \"9229:9229\" # Debug port exposed!\n environment:\n - DEBUG=true\n - NODE_ENV=development\n command: npm run dev # Nodemon in production!\n```\n\n**What Went Wrong**:\n1. Volume mount `.:/app` overwrote production code with empty directory\n2. Debug port `9229` exposed to internet\n3. `DEBUG=true` leaked sensitive information in logs\n4. `npm run dev` ran nodemon instead of production server\n5. No resource limits caused OOM under load\n\n**The Fix**: Separate development and production configurations that share a common base.\n\n---\n\n## 🧠 Mental Model: Environment Layering\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ COMPOSE FILE LAYERING │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ BASE LAYER (docker-compose.yml) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ • Service definitions │ │\n│ │ • Network topology │ │\n│ │ • Volume definitions │ │\n│ │ • Health checks │ │\n│ │ • Common environment variables │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ ↓ │\n│ DEV OVERRIDE (docker-compose.override.yml) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ • build context (instead of image) │ │\n│ │ • Volume mounts for hot reload │ │\n│ │ • Debug ports exposed │ │\n│ │ • DEV environment variables │ │\n│ │ • Local database ports exposed │ │\n│ │ • Dev tools (pgadmin, redis-commander) │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n│ PROD OVERRIDE (docker-compose.prod.yml) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ • Pre-built images │ │\n│ │ • No volume mounts (except data) │ │\n│ │ • Resource limits │ │\n│ │ • Production logging │ │\n│ │ • Security hardening │ │\n│ │ • Replicas for scaling │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n\nAuto-loaded: compose.yml + compose.override.yml\nProduction: compose.yml + compose.prod.yml\nStaging: compose.yml + compose.staging.yml\nTesting: compose.yml + compose.test.yml\n```\n\n---\n\n## 🔬 Deep Dive\n\n### 1. File Structure and Naming\n\n**Recommended Directory Structure**:\n\n```\nproject/\n├── docker-compose.yml # Base configuration\n├── docker-compose.override.yml # Development (auto-loaded)\n├── docker-compose.prod.yml # Production\n├── docker-compose.staging.yml # Staging (optional)\n├── docker-compose.test.yml # Testing\n├── .env # Default environment\n├── .env.example # Template for .env\n├── .env.production # Production env (gitignored)\n├── Dockerfile # Multi-stage Dockerfile\n└── docker/\n ├── nginx/\n │ └── nginx.conf\n └── scripts/\n └── entrypoint.sh\n```\n\n**Naming Conventions**:\n\n```bash\n# Compose V2 supports both naming styles\ndocker-compose.yml # Classic style\ncompose.yml # New style (Compose V2)\n\n# Override files\ndocker-compose.override.yml # Auto-loaded with base\ncompose.override.yml # Also auto-loaded\n\n# Environment-specific\ndocker-compose.prod.yml\ndocker-compose.staging.yml\ndocker-compose.test.yml\n```\n\n---\n\n### 2. Base Configuration (docker-compose.yml)\n\nThe base file contains everything common across all environments:\n\n```yaml\n# docker-compose.yml - Base configuration\n\nx-common: &common\n restart: unless-stopped\n logging:\n driver: json-file\n options:\n max-size: \"10m\"\n max-file: \"3\"\n\nservices:\n # ═══════════════════════════════════════════════════════════════\n # API SERVICE\n # ═══════════════════════════════════════════════════════════════\n api:\n <<: *common\n # Image specified in override files\n environment:\n - NODE_ENV=${NODE_ENV:-development}\n - PORT=8080\n - DATABASE_URL=${DATABASE_URL}\n - REDIS_URL=redis://redis:6379\n depends_on:\n db:\n condition: service_healthy\n redis:\n condition: service_started\n healthcheck:\n test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8080/health\"]\n interval: 30s\n timeout: 10s\n retries: 3\n start_period: 40s\n networks:\n - frontend\n - backend\n\n # ═══════════════════════════════════════════════════════════════\n # WORKER SERVICE\n # ═══════════════════════════════════════════════════════════════\n worker:\n <<: *common\n environment:\n - NODE_ENV=${NODE_ENV:-development}\n - DATABASE_URL=${DATABASE_URL}\n - REDIS_URL=redis://redis:6379\n depends_on:\n db:\n condition: service_healthy\n redis:\n condition: service_started\n networks:\n - backend\n\n # ═══════════════════════════════════════════════════════════════\n # DATABASE\n # ═══════════════════════════════════════════════════════════════\n db:\n <<: *common\n image: postgres:16-alpine\n environment:\n POSTGRES_USER: ${POSTGRES_USER:-app}\n POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Required}\n POSTGRES_DB: ${POSTGRES_DB:-app}\n volumes:\n - pgdata:/var/lib/postgresql/data\n healthcheck:\n test: [\"CMD-SHELL\", \"pg_isready -U ${POSTGRES_USER:-app}\"]\n interval: 10s\n timeout: 5s\n retries: 5\n networks:\n - backend\n\n # ═══════════════════════════════════════════════════════════════\n # REDIS\n # ═══════════════════════════════════════════════════════════════\n redis:\n <<: *common\n image: redis:7-alpine\n command: redis-server --appendonly yes\n volumes:\n - redisdata:/data\n healthcheck:\n test: [\"CMD\", \"redis-cli\", \"ping\"]\n interval: 10s\n timeout: 5s\n retries: 5\n networks:\n - backend\n\n# ═══════════════════════════════════════════════════════════════\n# NETWORKS\n# ═══════════════════════════════════════════════════════════════\nnetworks:\n frontend:\n driver: bridge\n backend:\n driver: bridge\n internal: true # No external access\n\n# ═══════════════════════════════════════════════════════════════\n# VOLUMES\n# ═══════════════════════════════════════════════════════════════\nvolumes:\n pgdata:\n redisdata:\n```\n\n---\n\n### 3. Development Override (docker-compose.override.yml)\n\nDevelopment focuses on productivity: hot reload, debugging, easy database access:\n\n```yaml\n# docker-compose.override.yml - Development configuration\n# Auto-loaded with docker-compose.yml\n\nservices:\n # ═══════════════════════════════════════════════════════════════\n # API - Development Mode\n # ═══════════════════════════════════════════════════════════════\n api:\n build:\n context: .\n target: development # Use dev stage from multi-stage Dockerfile\n volumes:\n # Hot reload: mount source code\n - .:/app\n # Preserve node_modules from container (don't overwrite with host's)\n - /app/node_modules\n ports:\n - \"8080:8080\" # API port\n - \"9229:9229\" # Debug port\n environment:\n - DEBUG=*\n - LOG_LEVEL=debug\n command: npm run dev # Nodemon or similar\n\n # ═══════════════════════════════════════════════════════════════\n # WORKER - Development Mode\n # ═══════════════════════════════════════════════════════════════\n worker:\n build:\n context: .\n target: development\n volumes:\n - .:/app\n - /app/node_modules\n environment:\n - DEBUG=*\n - LOG_LEVEL=debug\n command: npm run worker:dev\n\n # ═══════════════════════════════════════════════════════════════\n # DATABASE - Expose for local tools\n # ═══════════════════════════════════════════════════════════════\n db:\n ports:\n - \"5432:5432\" # Access from host\n environment:\n - POSTGRES_PASSWORD=devpassword # Simple dev password\n\n # ═══════════════════════════════════════════════════════════════\n # REDIS - Expose for local tools\n # ═══════════════════════════════════════════════════════════════\n redis:\n ports:\n - \"6379:6379\"\n\n # ═══════════════════════════════════════════════════════════════\n # DEV TOOLS (profiles ensure they're opt-in)\n # ═══════════════════════════════════════════════════════════════\n pgadmin:\n image: dpage/pgadmin4:latest\n environment:\n PGADMIN_DEFAULT_EMAIL: dev@local.dev\n PGADMIN_DEFAULT_PASSWORD: devpassword\n ports:\n - \"5050:80\"\n networks:\n - backend\n profiles:\n - tools\n\n redis-commander:\n image: rediscommander/redis-commander:latest\n environment:\n REDIS_HOSTS: local:redis:6379\n ports:\n - \"8081:8081\"\n networks:\n - backend\n profiles:\n - tools\n\n mailhog:\n image: mailhog/mailhog:latest\n ports:\n - \"1025:1025\" # SMTP\n - \"8025:8025\" # Web UI\n networks:\n - backend\n profiles:\n - tools\n```\n\n**Development .env file**:\n\n```bash\n# .env (development, committed to repo)\nNODE_ENV=development\nPOSTGRES_USER=app\nPOSTGRES_PASSWORD=devpassword\nPOSTGRES_DB=app\nDATABASE_URL=postgres://app:devpassword@db:5432/app\nLOG_LEVEL=debug\n```\n\n**Running Development**:\n\n```bash\n# Start everything (auto-loads override)\ndocker compose up\n\n# Start with dev tools\ndocker compose --profile tools up\n\n# Rebuild after dependency changes\ndocker compose up --build\n\n# View logs\ndocker compose logs -f api\n```\n\n---\n\n### 4. Production Override (docker-compose.prod.yml)\n\nProduction focuses on security, reliability, and performance:\n\n```yaml\n# docker-compose.prod.yml - Production configuration\n\nservices:\n # ═══════════════════════════════════════════════════════════════\n # API - Production Mode\n # ═══════════════════════════════════════════════════════════════\n api:\n image: ${REGISTRY}/api:${VERSION}\n # NO build context - use pre-built image\n # NO volumes - code baked into image\n # NO debug ports\n environment:\n - NODE_ENV=production\n - LOG_LEVEL=info\n - DEBUG= # Explicitly empty\n deploy:\n mode: replicated\n replicas: 2\n resources:\n limits:\n cpus: '1'\n memory: 512M\n reservations:\n cpus: '0.5'\n memory: 256M\n update_config:\n parallelism: 1\n delay: 10s\n order: start-first\n logging:\n driver: json-file\n options:\n max-size: \"50m\"\n max-file: \"5\"\n tag: \"{{.Name}}/{{.ID}}\"\n security_opt:\n - no-new-privileges:true\n read_only: true\n tmpfs:\n - /tmp:size=100M,mode=1777\n\n # ═══════════════════════════════════════════════════════════════\n # WORKER - Production Mode\n # ═══════════════════════════════════════════════════════════════\n worker:\n image: ${REGISTRY}/worker:${VERSION}\n environment:\n - NODE_ENV=production\n - LOG_LEVEL=info\n - CONCURRENCY=10\n deploy:\n mode: replicated\n replicas: 2\n resources:\n limits:\n cpus: '0.5'\n memory: 256M\n security_opt:\n - no-new-privileges:true\n\n # ═══════════════════════════════════════════════════════════════\n # DATABASE - Production hardening\n # ═══════════════════════════════════════════════════════════════\n db:\n # NO exposed ports in production\n ports: []\n deploy:\n resources:\n limits:\n cpus: '2'\n memory: 2G\n environment:\n # Password from secrets manager in real prod\n POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Required}\n\n # ═══════════════════════════════════════════════════════════════\n # REDIS - Production hardening\n # ═══════════════════════════════════════════════════════════════\n redis:\n ports: []\n command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru\n deploy:\n resources:\n limits:\n memory: 300M\n\n # ═══════════════════════════════════════════════════════════════\n # NGINX - Reverse Proxy (production only)\n # ═══════════════════════════════════════════════════════════════\n nginx:\n image: nginx:alpine\n ports:\n - \"80:80\"\n - \"443:443\"\n volumes:\n - ./docker/nginx/nginx.prod.conf:/etc/nginx/nginx.conf:ro\n - ./docker/nginx/ssl:/etc/nginx/ssl:ro\n depends_on:\n api:\n condition: service_healthy\n networks:\n - frontend\n deploy:\n resources:\n limits:\n cpus: '0.5'\n memory: 128M\n security_opt:\n - no-new-privileges:true\n read_only: true\n tmpfs:\n - /var/cache/nginx:size=100M\n - /var/run:size=10M\n```\n\n**Production .env.production file** (gitignored):\n\n```bash\n# .env.production (NOT committed, managed via secrets)\nNODE_ENV=production\nREGISTRY=registry.example.com\nVERSION=1.2.3\nPOSTGRES_USER=app_prod\nPOSTGRES_PASSWORD=\nPOSTGRES_DB=app_prod\nDATABASE_URL=postgres://app_prod:@db:5432/app_prod\n```\n\n**Running Production**:\n\n```bash\n# Explicitly specify files (don't auto-load override)\ndocker compose -f docker-compose.yml -f docker-compose.prod.yml up -d\n\n# Or set COMPOSE_FILE\nexport COMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml\ndocker compose up -d\n\n# Rolling update\ndocker compose pull\ndocker compose up -d --no-deps api\ndocker compose up -d --no-deps worker\n```\n\n---\n\n### 5. Multi-Stage Dockerfile\n\nSingle Dockerfile that serves both development and production:\n\n```dockerfile\n# Dockerfile\n\n# ═══════════════════════════════════════════════════════════════\n# BASE - Common dependencies\n# ═══════════════════════════════════════════════════════════════\nFROM node:20-alpine AS base\n\nWORKDIR /app\n\n# Install production dependencies only\nCOPY package*.json ./\nRUN npm ci --only=production\n\n# ═══════════════════════════════════════════════════════════════\n# DEVELOPMENT - Full dev environment\n# ═══════════════════════════════════════════════════════════════\nFROM node:20-alpine AS development\n\nWORKDIR /app\n\n# Install ALL dependencies (including devDependencies)\nCOPY package*.json ./\nRUN npm ci\n\n# Don't copy source - it will be mounted as volume\n# Install dev tools\nRUN npm install -g nodemon\n\n# Debug port\nEXPOSE 9229\n# App port\nEXPOSE 8080\n\nCMD [\"npm\", \"run\", \"dev\"]\n\n# ═══════════════════════════════════════════════════════════════\n# BUILD - Compile/bundle application\n# ═══════════════════════════════════════════════════════════════\nFROM node:20-alpine AS build\n\nWORKDIR /app\n\n# Need all deps for build\nCOPY package*.json ./\nRUN npm ci\n\n# Copy source and build\nCOPY . .\nRUN npm run build\n\n# ═══════════════════════════════════════════════════════════════\n# PRODUCTION - Minimal runtime\n# ═══════════════════════════════════════════════════════════════\nFROM node:20-alpine AS production\n\nWORKDIR /app\n\n# Non-root user\nRUN addgroup -g 1001 -S nodejs && \\\n adduser -S nodejs -u 1001\n\n# Copy production deps from base\nCOPY --from=base /app/node_modules ./node_modules\n\n# Copy built application from build stage\nCOPY --from=build /app/dist ./dist\nCOPY --from=build /app/package.json ./\n\n# Security hardening\nUSER nodejs\n\nEXPOSE 8080\n\n# Health check\nHEALTHCHECK --interval=30s --timeout=10s --retries=3 \\\n CMD wget -q --spider http://localhost:8080/health || exit 1\n\nCMD [\"node\", \"dist/server.js\"]\n```\n\n**Usage in compose**:\n\n```yaml\n# Development (override)\napi:\n build:\n context: .\n target: development\n\n# Production\napi:\n image: registry/api:${VERSION}\n # Or build:\n build:\n context: .\n target: production\n```\n\n---\n\n### 6. Environment-Specific Secrets\n\n**Development**: Simple, committed secrets (okay for local dev):\n\n```yaml\n# docker-compose.override.yml\nservices:\n api:\n environment:\n - JWT_SECRET=dev-secret-not-for-production\n - API_KEY=dev-api-key\n```\n\n**Production**: External secrets (never in compose files):\n\n```yaml\n# docker-compose.prod.yml\nservices:\n api:\n environment:\n # From environment variable (CI/CD injects)\n - JWT_SECRET=${JWT_SECRET:?Required}\n - API_KEY=${API_KEY:?Required}\n\n # Or using Docker secrets (Swarm mode)\n secrets:\n - jwt_secret\n - api_key\n\nsecrets:\n jwt_secret:\n external: true\n api_key:\n external: true\n```\n\n**Docker Secrets Pattern**:\n\n```bash\n# Create secrets (Swarm mode)\necho \"super-secret-jwt-key\" | docker secret create jwt_secret -\n\n# In application, read from /run/secrets/\nconst jwtSecret = fs.readFileSync('/run/secrets/jwt_secret', 'utf8').trim();\n```\n\n**Environment Variables from File**:\n\n```yaml\n# docker-compose.prod.yml\nservices:\n api:\n env_file:\n - .env.production\n```\n\n---\n\n### 7. Testing Configuration\n\nTesting needs isolation and speed:\n\n```yaml\n# docker-compose.test.yml\n\nservices:\n api:\n build:\n context: .\n target: development\n environment:\n - NODE_ENV=test\n - DATABASE_URL=postgres://test:test@db-test:5432/test\n - REDIS_URL=redis://redis-test:6379\n depends_on:\n db-test:\n condition: service_healthy\n redis-test:\n condition: service_started\n networks:\n - test\n\n # Separate test databases\n db-test:\n image: postgres:16-alpine\n environment:\n POSTGRES_USER: test\n POSTGRES_PASSWORD: test\n POSTGRES_DB: test\n healthcheck:\n test: pg_isready -U test\n interval: 5s\n timeout: 5s\n retries: 5\n # tmpfs for speed\n tmpfs:\n - /var/lib/postgresql/data\n networks:\n - test\n\n redis-test:\n image: redis:7-alpine\n networks:\n - test\n\n # Test runner\n test-runner:\n build:\n context: .\n target: development\n command: npm test\n environment:\n - NODE_ENV=test\n - DATABASE_URL=postgres://test:test@db-test:5432/test\n depends_on:\n api:\n condition: service_healthy\n networks:\n - test\n\nnetworks:\n test:\n driver: bridge\n```\n\n**Running Tests**:\n\n```bash\n# Run tests in isolated environment\ndocker compose -f docker-compose.yml -f docker-compose.test.yml run --rm test-runner\n\n# CI/CD pipeline\ndocker compose -f docker-compose.yml -f docker-compose.test.yml up -d db-test redis-test\ndocker compose -f docker-compose.yml -f docker-compose.test.yml run --rm test-runner\ndocker compose -f docker-compose.yml -f docker-compose.test.yml down -v\n```\n\n---\n\n### 8. Complete Example: Makefile for Environment Management\n\n```makefile\n# Makefile\n\n.PHONY: dev dev-tools prod test clean\n\n# Development (auto-loads override)\ndev:\n\tdocker compose up\n\ndev-build:\n\tdocker compose up --build\n\ndev-tools:\n\tdocker compose --profile tools up\n\n# Production\nprod:\n\tdocker compose -f docker-compose.yml -f docker-compose.prod.yml up -d\n\nprod-build:\n\tdocker compose -f docker-compose.yml -f docker-compose.prod.yml build\n\nprod-deploy:\n\tdocker compose -f docker-compose.yml -f docker-compose.prod.yml pull\n\tdocker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --no-deps api worker\n\n# Testing\ntest:\n\tdocker compose -f docker-compose.yml -f docker-compose.test.yml run --rm test-runner\n\tdocker compose -f docker-compose.yml -f docker-compose.test.yml down -v\n\ntest-watch:\n\tdocker compose -f docker-compose.yml -f docker-compose.test.yml run --rm test-runner npm run test:watch\n\n# Cleanup\nclean:\n\tdocker compose down -v --remove-orphans\n\tdocker compose -f docker-compose.yml -f docker-compose.test.yml down -v --remove-orphans\n\n# Logs\nlogs:\n\tdocker compose logs -f\n\nlogs-api:\n\tdocker compose logs -f api\n\n# Shell access\nshell-api:\n\tdocker compose exec api sh\n\nshell-db:\n\tdocker compose exec db psql -U app\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Same Dockerfile CMD for Dev and Prod\n\n```dockerfile\n# ❌ Wrong: Production runs dev server\nCMD [\"npm\", \"run\", \"dev\"]\n\n# ✅ Right: Different targets\n# Development stage\nFROM base AS development\nCMD [\"npm\", \"run\", \"dev\"]\n\n# Production stage\nFROM base AS production\nCMD [\"npm\", \"start\"]\n```\n\n### Mistake 2: Volume Mount in Production\n\n```yaml\n# ❌ Wrong: Mounts empty host directory over container code\nservices:\n api:\n volumes:\n - .:/app # Fine in dev, disaster in prod\n\n# ✅ Right: Only in development override\n# docker-compose.yml: No volume mounts\n# docker-compose.override.yml:\nservices:\n api:\n volumes:\n - .:/app\n```\n\n### Mistake 3: Exposing Debug Ports in Production\n\n```yaml\n# ❌ Wrong: Debug port in base file\nservices:\n api:\n ports:\n - \"8080:8080\"\n - \"9229:9229\" # Debug - should only be in dev!\n\n# ✅ Right: Debug port only in dev override\n# docker-compose.yml:\nservices:\n api:\n ports:\n - \"8080:8080\"\n\n# docker-compose.override.yml:\nservices:\n api:\n ports:\n - \"9229:9229\" # Add debug port for dev\n```\n\n### Mistake 4: Hardcoded Development Settings\n\n```yaml\n# ❌ Wrong: Dev settings in base\nservices:\n api:\n environment:\n - DEBUG=true\n - LOG_LEVEL=debug\n\n# ✅ Right: Use environment variables with defaults\nservices:\n api:\n environment:\n - DEBUG=${DEBUG:-false}\n - LOG_LEVEL=${LOG_LEVEL:-info}\n```\n\n### Mistake 5: No Resource Limits in Production\n\n```yaml\n# ❌ Wrong: No limits - one service can starve others\nservices:\n api:\n image: myapi\n\n# ✅ Right: Always set limits in production\nservices:\n api:\n image: myapi\n deploy:\n resources:\n limits:\n cpus: '1'\n memory: 512M\n```\n\n---\n\n## 🐛 Debug This\n\nDevelopment works, production fails:\n\n```yaml\n# docker-compose.yml\nservices:\n api:\n build: .\n environment:\n - DATABASE_URL=${DATABASE_URL}\n volumes:\n - .:/app\n\n# docker-compose.prod.yml\nservices:\n api:\n image: myapi:${VERSION}\n```\n\n```bash\n# Development: works\n$ docker compose up\napi_1 | Server running on port 8080\n\n# Production: fails\n$ docker compose -f docker-compose.yml -f docker-compose.prod.yml up\napi_1 | Error: Cannot find module '/app/server.js'\n```\n\nWhy does production fail?\n\n

Click to reveal analysis

\n\n**Problem**: Volume mount from base file persists into production!\n\n```yaml\n# docker-compose.yml has:\nvolumes:\n - .:/app # This overrides the container's /app\n\n# Production server likely doesn't have the source code at .\n# The volume mount overwrites the image's /app with an empty or wrong directory\n```\n\n**Compose merge rules**: Arrays (like volumes) are **appended**, not replaced. The production override doesn't remove the volume mount.\n\n**Fix Option 1**: Move volume to dev-only override:\n\n```yaml\n# docker-compose.yml (base)\nservices:\n api:\n environment:\n - DATABASE_URL=${DATABASE_URL}\n # NO volumes here\n\n# docker-compose.override.yml (dev)\nservices:\n api:\n build: .\n volumes:\n - .:/app\n - /app/node_modules\n\n# docker-compose.prod.yml\nservices:\n api:\n image: myapi:${VERSION}\n # No volumes needed\n```\n\n**Fix Option 2**: Explicitly clear volumes in prod:\n\n```yaml\n# docker-compose.prod.yml\nservices:\n api:\n image: myapi:${VERSION}\n volumes: [] # Explicitly empty\n```\n\nNote: `volumes: []` doesn't work in all compose versions. Best practice is to not have dev-only settings in the base file.\n\n

\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Basic Environment Split\nTake this single compose file and split it into base + dev override:\n```yaml\nservices:\n api:\n build: .\n volumes:\n - .:/app\n ports:\n - \"3000:3000\"\n - \"9229:9229\"\n environment:\n - DEBUG=true\n command: npm run dev\n```\n\n### Exercise 2: Production Hardening\nCreate a `docker-compose.prod.yml` that:\n- Uses pre-built images\n- Adds resource limits\n- Enables read-only filesystem\n- Removes debug settings\n- Adds nginx as reverse proxy\n\n### Exercise 3: Multi-Stage Dockerfile\nCreate a multi-stage Dockerfile with:\n- `development` target with all dev tools\n- `build` target that compiles the app\n- `production` target that's minimal and secure\n\n### Exercise 4: Test Configuration\nCreate `docker-compose.test.yml` that:\n- Uses separate test database (tmpfs for speed)\n- Runs tests in isolation\n- Cleans up after itself\n\n### Exercise 5: Secrets Management\nImplement different secret handling for:\n- Development: Simple env vars\n- Production: Docker secrets or external secrets manager\n\n---\n\n## 🎤 Interview Questions\n\n### Q1: How do you structure compose files for multiple environments?\n\n**Answer**: Use layered compose files with automatic and explicit loading:\n\n```\nproject/\n├── docker-compose.yml # Base (always loaded)\n├── docker-compose.override.yml # Dev (auto-loaded locally)\n├── docker-compose.prod.yml # Production\n├── docker-compose.test.yml # Testing\n```\n\n**Loading behavior**:\n```bash\n# Development (auto-loads override)\ndocker compose up\n\n# Production (explicit files)\ndocker compose -f docker-compose.yml -f docker-compose.prod.yml up\n\n# Or via environment variable\nCOMPOSE_FILE=docker-compose.yml:docker-compose.prod.yml docker compose up\n```\n\n**Key principles**:\n1. Base file: Service definitions, networks, health checks\n2. Dev override: Build context, volume mounts, debug ports\n3. Prod override: Images, resource limits, security hardening\n\n---\n\n### Q2: What should be in the base compose file vs environment-specific overrides?\n\n**Answer**:\n\n**Base file (docker-compose.yml)**:\n- Service names and structure\n- Network topology\n- Volume definitions (without mounts)\n- Health checks\n- Dependencies\n- Environment variables using `${VAR:-default}` pattern\n\n**Development override**:\n- `build:` context (build from source)\n- Volume mounts for hot reload\n- Debug ports (9229, etc.)\n- Dev tools (pgadmin, etc.)\n- Database ports exposed\n- `DEBUG=true`, verbose logging\n\n**Production override**:\n- `image:` (pre-built images)\n- Resource limits\n- Security options (read_only, no-new-privileges)\n- Production logging config\n- Replicas for scaling\n- Reverse proxy configuration\n\n---\n\n### Q3: How do you handle secrets differently in development vs production?\n\n**Answer**: Different strategies for different security needs:\n\n**Development** (convenience over security):\n```yaml\n# docker-compose.override.yml\nservices:\n api:\n environment:\n - JWT_SECRET=dev-not-secret\n - DB_PASSWORD=devpassword\n```\n\n**Production** (security first):\n\nOption 1 - Environment variables from CI/CD:\n```yaml\n# docker-compose.prod.yml\nservices:\n api:\n environment:\n - JWT_SECRET=${JWT_SECRET:?Required}\n```\n\nOption 2 - Docker secrets (Swarm):\n```yaml\nservices:\n api:\n secrets:\n - jwt_secret\n\nsecrets:\n jwt_secret:\n external: true\n```\n\nOption 3 - External secrets manager:\n```yaml\nservices:\n api:\n environment:\n # App fetches from vault/AWS Secrets Manager at runtime\n - VAULT_ADDR=https://vault:8200\n - VAULT_ROLE=api\n```\n\n**Best practice**: Never commit production secrets. Use CI/CD secret injection or external secrets management.\n\n---\n\n### Q4: How do you ensure development environment matches production?\n\n**Answer**: Environment parity with controlled differences:\n\n1. **Same base services**:\n```yaml\n# docker-compose.yml\nservices:\n db:\n image: postgres:16-alpine # Same version\n redis:\n image: redis:7-alpine # Same version\n```\n\n2. **Same network topology**:\n```yaml\nnetworks:\n frontend:\n backend:\n internal: true # Same isolation\n```\n\n3. **Multi-stage Dockerfile**:\n```dockerfile\n# Same base, different targets\nFROM node:20-alpine AS base\n# ...\nFROM base AS development\nFROM base AS production\n```\n\n4. **Controlled differences** (documented):\n- Dev: Volume mounts, debug ports, verbose logging\n- Prod: Resource limits, security hardening, monitoring\n\n5. **Same configurations**:\n```yaml\n# Share health checks, timeouts, retry policies\nx-healthcheck: &healthcheck\n interval: 30s\n timeout: 10s\n retries: 3\n```\n\n---\n\n### Q5: How would you debug an issue that only happens in production compose setup?\n\n**Answer**: Systematic approach to reproduce and diagnose:\n\n1. **Run production config locally**:\n```bash\n# Use prod config without prod secrets\ndocker compose -f docker-compose.yml -f docker-compose.prod.yml \\\n --env-file .env.local up\n```\n\n2. **Compare configurations**:\n```bash\n# Generate resolved config\ndocker compose config > dev-resolved.yml\ndocker compose -f docker-compose.yml -f docker-compose.prod.yml config > prod-resolved.yml\ndiff dev-resolved.yml prod-resolved.yml\n```\n\n3. **Check for common issues**:\n- Volume mounts persisting from base (overwriting image contents)\n- Missing environment variables (required in prod)\n- Resource limits causing OOM\n- Network isolation blocking communication\n- Read-only filesystem blocking temp files\n\n4. **Inspect running containers**:\n```bash\ndocker compose exec api env # Check environment\ndocker compose exec api ls -la /app # Check filesystem\ndocker compose logs api # Check logs\n```\n\n5. **Add debug temporarily**:\n```yaml\n# Temporary debug override\nservices:\n api:\n environment:\n - LOG_LEVEL=debug\n # Temporarily remove read_only for debugging\n read_only: false\n```\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### File Structure\n```\ndocker-compose.yml # Base (services, networks, volumes)\ndocker-compose.override.yml # Dev (auto-loaded)\ndocker-compose.prod.yml # Production (explicit)\n```\n\n### Development Config\n- Build from source\n- Volume mounts for hot reload\n- Debug ports exposed\n- Databases accessible\n- Verbose logging\n\n### Production Config\n- Pre-built images\n- No volume mounts (except data)\n- Resource limits\n- Security hardening\n- Minimal logging\n\n### Running Commands\n```bash\n# Dev (auto-loads override)\ndocker compose up\n\n# Prod (explicit files)\ndocker compose -f docker-compose.yml -f docker-compose.prod.yml up\n```\n\n---\n\n## 📋 Quick Reference\n\n```yaml\n# Base: Use variables with defaults\nenvironment:\n - NODE_ENV=${NODE_ENV:-development}\n\n# Dev: Build and mount\nbuild:\n context: .\n target: development\nvolumes:\n - .:/app\n\n# Prod: Image and limits\nimage: registry/app:${VERSION}\ndeploy:\n resources:\n limits:\n memory: 512M\nread_only: true\nsecurity_opt:\n - no-new-privileges:true\n```\n\n---\n\n## 📅 Review Schedule\n\n- **Day 1**: Split existing compose into base + override\n- **Day 3**: Add production override with security\n- **Day 7**: Implement multi-stage Dockerfile\n- **Day 14**: Set up complete dev/test/prod pipeline\n- **Day 30**: Audit production configs for security\n\n---\n\n## 📚 Series Navigation\n\n- **Previous**: [Part 15 - Service Dependencies & Orchestration](/articles/docker-part-15-dependencies)\n- **Next**: [Part 17 - CI/CD Pipeline Patterns](/articles/docker-part-17-cicd)\n- **Index**: [Docker Compendium Series](/series/docker-compendium)","featuredImage","isPublished","isFeatured","subscriptionType","BASIC","publishedAt","2026-06-10T07:19:25.546187Z","createdAt","2026-06-10T07:19:25.546800Z","updatedAt","2026-06-16T21:26:56.086200Z","readingTimeMinutes",15,"estimatedReadingTime","authorId","authorUsername","tomasz_main","categoryName","Devops","readingTime","seriesId","seriesOrder",16,"author",{"_34":16,"_96":59,"_97":98,"_99":100,"_101":-5,"_102":-5,"_103":-5,"_104":-5,"_105":-5},"category",{"_34":92,"_74":61,"_38":93,"_77":94,"_95":-5},"tags",[80,81,82],"series",{"_34":16,"_74":75,"_38":76,"_77":78,"_79":24},"name","Docker Compendium: From Practitioner to Expert","docker-compendium","description","Master Docker internals, optimization patterns, and production best practices. For developers who use Docker daily but want deeper understanding of what's happening under the hood. Not a beginner series - we assume you can write Dockerfiles and use docker-compose. Learn what namespaces and cgroups really do, why your builds are slow, how to debug containers that crash, and how to run Docker in production like a pro.\n","isActive",{"_34":89,"_74":90,"_38":91},{"_34":86,"_74":87,"_38":88},{"_34":83,"_74":84,"_38":85},216,"Production","production",213,"Docker","docker",242,"Compose","compose",7,"devops","Category for Devops","colorHex","username","firstName","Tomasz","lastName","Jarosz","profilePicture","bio","website","githubUrl","linkedinUrl",68,"Production Deployment Patterns","docker-part-18-deployment","Rolling updates, blue-green, zero-downtime","## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| **Difficulty** | 🟠 Advanced |\n| **Prerequisites** | Part 15 (Dependencies), Part 17 (CI/CD) |\n| **Patterns** | Rolling, Blue-Green, Canary |\n| **Time Investment** | 26 minutes read + 90 minutes practice |\n| **Payoff** | Zero-downtime deployments, confident rollbacks |\n\n---\n\n## 🎯 What You'll Learn\n\nAfter this article, you'll be able to:\n\n1. **Implement rolling updates** with zero downtime\n2. **Execute blue-green deployments** for instant rollbacks\n3. **Run canary deployments** to test with real traffic\n4. **Handle rollbacks** quickly and safely\n5. **Manage database migrations** during deployments\n\n---\n\n## 🔥 Production Story: The Friday Afternoon Deployment\n\n**The Setup**: 4:30 PM Friday. \"Quick hotfix, just push it.\" No testing, direct `docker compose up -d`.\n\n**What Happened**:\n```bash\n$ docker compose up -d\nRecreating api_1...\n# 100% of traffic goes to new container\n# New container fails health check\n# 30 seconds of 502 errors before old container comes back\n```\n\n**The Damage**:\n- 30 seconds of complete outage\n- 1,000+ failed requests\n- Angry customers over the weekend\n\n**The Root Cause**: No deployment strategy. `docker compose up -d` stops old container before new one is ready.\n\n**The Fix**:\n```yaml\nservices:\n api:\n deploy:\n update_config:\n order: start-first # Start new BEFORE stopping old\n failure_action: rollback\n parallelism: 1\n delay: 10s\n```\n\n**Result**: New deployments start the new container, wait for it to be healthy, then stop the old one. Zero downtime.\n\n---\n\n## 🧠 Mental Model: Deployment Strategies\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ DEPLOYMENT STRATEGIES │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ ROLLING UPDATE (Default, safest) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ v1 ████████████████████ │ │\n│ │ v2 ████████████████████ │ │\n│ │ ↑ start new ↑ stop old │ │\n│ │ start-first: new healthy before old stops │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n│ BLUE-GREEN (Instant switch) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ Blue (v1) ████████████████ │ │\n│ │ Green (v2) ████████████████████████████ │ │\n│ │ ↑ switch LB │ │\n│ │ Both running, traffic switches instantly │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n│ CANARY (Gradual rollout) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ v1 ████████████████ (90%) │ │\n│ │ v2 ██ (10%) ████ (25%) ████████████ (100%) │ │\n│ │ ↑ test with few ↑ increase ↑ full rollout │ │\n│ │ Gradually shift traffic if metrics OK │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n│ RECREATE (Downtime, simple) │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ v1 ████████████████ │ │\n│ │ (downtime) │ │\n│ │ v2 ████████████████████ │ │\n│ │ ↑ stop all ↑ start all │ │\n│ │ Simple but has downtime │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## 🔬 Deep Dive\n\n### 1. Rolling Updates with Docker Compose\n\n**Basic Rolling Update**:\n\n```yaml\n# docker-compose.yml\nservices:\n api:\n image: myapp:${VERSION:-latest}\n deploy:\n replicas: 3\n update_config:\n parallelism: 1 # Update 1 container at a time\n delay: 10s # Wait 10s between updates\n order: start-first # Start new before stopping old\n failure_action: rollback\n rollback_config:\n parallelism: 1\n delay: 5s\n healthcheck:\n test: curl -f http://localhost:8080/health\n interval: 10s\n timeout: 5s\n retries: 3\n start_period: 30s\n```\n\n**Deployment Script**:\n\n```bash\n#!/bin/bash\n# deploy-rolling.sh\n\nset -e\n\nVERSION=$1\nif [ -z \"$VERSION\" ]; then\n echo \"Usage: $0 \"\n exit 1\nfi\n\nexport VERSION\n\necho \"Deploying version: $VERSION\"\n\n# Pull new image first\ndocker compose pull api\n\n# Rolling update\ndocker compose up -d --no-recreate --scale api=3 api\n\n# Wait for health checks\necho \"Waiting for containers to be healthy...\"\nfor i in {1..30}; do\n HEALTHY=$(docker compose ps --format json | jq -r 'select(.Health == \"healthy\")' | wc -l)\n if [ \"$HEALTHY\" -ge 3 ]; then\n echo \"All containers healthy!\"\n break\n fi\n echo \"Healthy: $HEALTHY/3\"\n sleep 5\ndone\n\n# Verify deployment\ndocker compose ps\n```\n\n**Manual Rolling Update** (without Swarm):\n\n```bash\n#!/bin/bash\n# manual-rolling-update.sh\n\nVERSION=$1\nSERVICE=api\n\n# Get current container count\nCURRENT_COUNT=$(docker compose ps -q $SERVICE | wc -l)\n\necho \"Scaling up with new version...\"\nVERSION=$VERSION docker compose up -d --no-deps --scale $SERVICE=$((CURRENT_COUNT + 1)) --no-recreate $SERVICE\n\necho \"Waiting for new container to be healthy...\"\nsleep 30\n\necho \"Removing old containers one by one...\"\nOLD_CONTAINERS=$(docker compose ps -q $SERVICE | head -n $CURRENT_COUNT)\nfor CONTAINER in $OLD_CONTAINERS; do\n echo \"Stopping $CONTAINER...\"\n docker stop $CONTAINER\n docker rm $CONTAINER\n sleep 10\ndone\n\necho \"Deployment complete\"\ndocker compose ps\n```\n\n---\n\n### 2. Blue-Green Deployment\n\n**Architecture**:\n\n```\n ┌──────────────┐\n │ Nginx/LB │\n └──────┬───────┘\n │\n ┌────────────┴────────────┐\n │ │\n ┌─────┴─────┐ ┌─────┴─────┐\n │ Blue │ │ Green │\n │ (live) │ │ (new) │\n │ port 80 │ │ port 8080 │\n └───────────┘ └───────────┘\n```\n\n**Compose Configuration**:\n\n```yaml\n# docker-compose.blue-green.yml\n\nservices:\n nginx:\n image: nginx:alpine\n ports:\n - \"80:80\"\n volumes:\n - ./nginx.conf:/etc/nginx/nginx.conf:ro\n depends_on:\n - blue\n - green\n networks:\n - frontend\n\n blue:\n image: myapp:${BLUE_VERSION:-latest}\n environment:\n - COLOR=blue\n networks:\n - frontend\n - backend\n deploy:\n replicas: 2\n\n green:\n image: myapp:${GREEN_VERSION:-latest}\n environment:\n - COLOR=green\n networks:\n - frontend\n - backend\n deploy:\n replicas: 2\n\nnetworks:\n frontend:\n backend:\n```\n\n**Nginx Configuration**:\n\n```nginx\n# nginx.conf\nupstream backend {\n # Active color controlled by env or config reload\n server blue:8080;\n # server green:8080;\n}\n\nserver {\n listen 80;\n\n location / {\n proxy_pass http://backend;\n proxy_set_header Host $host;\n proxy_set_header X-Real-IP $remote_addr;\n }\n\n location /health {\n return 200 'OK';\n }\n}\n```\n\n**Blue-Green Deployment Script**:\n\n```bash\n#!/bin/bash\n# deploy-blue-green.sh\n\nset -e\n\nVERSION=$1\nCURRENT_COLOR=$(cat .current-color 2>/dev/null || echo \"blue\")\n\nif [ \"$CURRENT_COLOR\" = \"blue\" ]; then\n NEW_COLOR=\"green\"\n export GREEN_VERSION=$VERSION\nelse\n NEW_COLOR=\"blue\"\n export BLUE_VERSION=$VERSION\nfi\n\necho \"Current: $CURRENT_COLOR, Deploying to: $NEW_COLOR\"\n\n# 1. Deploy new version to inactive color\necho \"Starting $NEW_COLOR with version $VERSION...\"\ndocker compose -f docker-compose.blue-green.yml up -d $NEW_COLOR\n\n# 2. Wait for new deployment to be healthy\necho \"Waiting for $NEW_COLOR to be healthy...\"\nfor i in {1..30}; do\n if docker compose -f docker-compose.blue-green.yml exec $NEW_COLOR \\\n curl -sf http://localhost:8080/health > /dev/null 2>&1; then\n echo \"$NEW_COLOR is healthy!\"\n break\n fi\n echo \"Waiting... ($i/30)\"\n sleep 5\ndone\n\n# 3. Run smoke tests against new color\necho \"Running smoke tests...\"\ndocker compose -f docker-compose.blue-green.yml exec $NEW_COLOR \\\n npm run test:smoke || {\n echo \"Smoke tests failed! Aborting.\"\n exit 1\n}\n\n# 4. Switch traffic (update nginx config)\necho \"Switching traffic to $NEW_COLOR...\"\nsed -i \"s/server $CURRENT_COLOR/# server $CURRENT_COLOR/g\" nginx.conf\nsed -i \"s/# server $NEW_COLOR/server $NEW_COLOR/g\" nginx.conf\ndocker compose -f docker-compose.blue-green.yml exec nginx nginx -s reload\n\n# 5. Save current color\necho $NEW_COLOR > .current-color\n\n# 6. Optionally stop old color (keep for quick rollback)\necho \"Deployment complete. Old ($CURRENT_COLOR) still running for rollback.\"\necho \"Run './stop-old.sh' to remove old deployment.\"\n```\n\n**Instant Rollback**:\n\n```bash\n#!/bin/bash\n# rollback-blue-green.sh\n\nCURRENT_COLOR=$(cat .current-color)\nif [ \"$CURRENT_COLOR\" = \"blue\" ]; then\n ROLLBACK_COLOR=\"green\"\nelse\n ROLLBACK_COLOR=\"blue\"\nfi\n\necho \"Rolling back from $CURRENT_COLOR to $ROLLBACK_COLOR...\"\n\n# Just switch nginx upstream\nsed -i \"s/server $CURRENT_COLOR/# server $CURRENT_COLOR/g\" nginx.conf\nsed -i \"s/# server $ROLLBACK_COLOR/server $ROLLBACK_COLOR/g\" nginx.conf\ndocker compose -f docker-compose.blue-green.yml exec nginx nginx -s reload\n\necho $ROLLBACK_COLOR > .current-color\necho \"Rolled back to $ROLLBACK_COLOR\"\n```\n\n---\n\n### 3. Canary Deployment\n\n**Nginx with Weighted Traffic**:\n\n```nginx\n# nginx.conf for canary\nupstream backend {\n # 90% to stable\n server stable:8080 weight=9;\n # 10% to canary\n server canary:8080 weight=1;\n}\n\nserver {\n listen 80;\n\n location / {\n proxy_pass http://backend;\n }\n}\n```\n\n**Compose for Canary**:\n\n```yaml\n# docker-compose.canary.yml\n\nservices:\n nginx:\n image: nginx:alpine\n ports:\n - \"80:80\"\n volumes:\n - ./nginx-canary.conf:/etc/nginx/nginx.conf:ro\n networks:\n - frontend\n\n stable:\n image: myapp:${STABLE_VERSION:-latest}\n deploy:\n replicas: 9\n networks:\n - frontend\n - backend\n\n canary:\n image: myapp:${CANARY_VERSION:-latest}\n deploy:\n replicas: 1\n networks:\n - frontend\n - backend\n\nnetworks:\n frontend:\n backend:\n```\n\n**Canary Deployment Script**:\n\n```bash\n#!/bin/bash\n# deploy-canary.sh\n\nset -e\n\nVERSION=$1\nCANARY_PERCENT=${2:-10}\n\nexport CANARY_VERSION=$VERSION\n\necho \"Starting canary deployment: $VERSION ($CANARY_PERCENT%)\"\n\n# 1. Deploy canary\ndocker compose -f docker-compose.canary.yml up -d canary\n\n# 2. Wait for healthy\necho \"Waiting for canary to be healthy...\"\nsleep 30\n\n# 3. Update nginx weights\nSTABLE_WEIGHT=$((100 - CANARY_PERCENT))\ncat > nginx-canary.conf << EOF\nupstream backend {\n server stable:8080 weight=$STABLE_WEIGHT;\n server canary:8080 weight=$CANARY_PERCENT;\n}\nserver {\n listen 80;\n location / { proxy_pass http://backend; }\n}\nEOF\n\ndocker compose -f docker-compose.canary.yml exec nginx nginx -s reload\n\necho \"Canary deployed with $CANARY_PERCENT% traffic\"\necho \"Monitor metrics. Run './promote-canary.sh' or './rollback-canary.sh'\"\n```\n\n**Promote or Rollback**:\n\n```bash\n#!/bin/bash\n# promote-canary.sh\n\nCANARY_VERSION=$(docker inspect canary --format '{{.Config.Image}}' | cut -d: -f2)\n\necho \"Promoting canary version: $CANARY_VERSION\"\n\n# Update stable to canary version\nexport STABLE_VERSION=$CANARY_VERSION\ndocker compose -f docker-compose.canary.yml up -d stable\n\n# Remove canary\ndocker compose -f docker-compose.canary.yml stop canary\ndocker compose -f docker-compose.canary.yml rm -f canary\n\n# Reset nginx to stable only\ncat > nginx-canary.conf << EOF\nupstream backend {\n server stable:8080;\n}\nserver {\n listen 80;\n location / { proxy_pass http://backend; }\n}\nEOF\n\ndocker compose -f docker-compose.canary.yml exec nginx nginx -s reload\n\necho \"Canary promoted to stable\"\n```\n\n---\n\n### 4. Handling Database Migrations\n\n**Migration Strategies**:\n\n```\n┌────────────────────────────────────────────────────────────┐\n│ MIGRATION PATTERNS │\n├────────────────────────────────────────────────────────────┤\n│ │\n│ 1. BACKWARDS COMPATIBLE (preferred) │\n│ ┌─────────────────────────────────────────────────┐ │\n│ │ v1: Uses column 'name' │ │\n│ │ Migration: Add column 'full_name', copy data │ │\n│ │ v2: Uses 'full_name', ignores 'name' │ │\n│ │ Later: Drop 'name' column │ │\n│ └─────────────────────────────────────────────────┘ │\n│ │\n│ 2. EXPAND-CONTRACT │\n│ ┌─────────────────────────────────────────────────┐ │\n│ │ Expand: Add new column/table │ │\n│ │ Deploy: v2 writes to both old and new │ │\n│ │ Migrate: Backfill old data │ │\n│ │ Contract: v3 only uses new, drop old │ │\n│ └─────────────────────────────────────────────────┘ │\n│ │\n│ 3. FEATURE FLAGS │\n│ ┌─────────────────────────────────────────────────┐ │\n│ │ Deploy code with flag off │ │\n│ │ Run migration │ │\n│ │ Enable flag for small % of users │ │\n│ │ Roll out to 100% │ │\n│ └─────────────────────────────────────────────────┘ │\n│ │\n└────────────────────────────────────────────────────────────┘\n```\n\n**Migration Service in Compose**:\n\n```yaml\nservices:\n migrate:\n image: myapp:${VERSION}\n command: npm run migrate\n environment:\n - DATABASE_URL=${DATABASE_URL}\n depends_on:\n db:\n condition: service_healthy\n restart: \"no\"\n profiles:\n - migrate\n\n api:\n image: myapp:${VERSION}\n depends_on:\n db:\n condition: service_healthy\n```\n\n**Deployment with Migration**:\n\n```bash\n#!/bin/bash\n# deploy-with-migration.sh\n\nVERSION=$1\nexport VERSION\n\n# 1. Run migrations first\necho \"Running migrations...\"\ndocker compose --profile migrate run --rm migrate\n\n# Check migration success\nif [ $? -ne 0 ]; then\n echo \"Migration failed! Aborting deployment.\"\n exit 1\nfi\n\n# 2. Deploy new version\necho \"Deploying application...\"\ndocker compose up -d api\n\necho \"Deployment complete\"\n```\n\n**Reversible Migration Example**:\n\n```sql\n-- migrations/001_add_user_email.sql\n-- Up\nALTER TABLE users ADD COLUMN email VARCHAR(255);\nCREATE INDEX idx_users_email ON users(email);\n\n-- Down\nDROP INDEX idx_users_email;\nALTER TABLE users DROP COLUMN email;\n```\n\n---\n\n### 5. Health Checks and Readiness\n\n**Liveness vs Readiness**:\n\n```yaml\nservices:\n api:\n healthcheck:\n # This is liveness - is the process running?\n test: curl -f http://localhost:8080/health/live\n interval: 10s\n timeout: 5s\n retries: 3\n```\n\n**Application Health Endpoints**:\n\n```javascript\n// Express.js health endpoints\n\n// Liveness: Is the process alive?\napp.get('/health/live', (req, res) => {\n res.json({ status: 'alive' });\n});\n\n// Readiness: Can we serve traffic?\napp.get('/health/ready', async (req, res) => {\n const checks = {\n database: await checkDatabase(),\n redis: await checkRedis(),\n dependencies: await checkExternalServices()\n };\n\n const allHealthy = Object.values(checks).every(c => c.healthy);\n\n res.status(allHealthy ? 200 : 503).json({\n status: allHealthy ? 'ready' : 'not ready',\n checks\n });\n});\n\n// Full health: Detailed status\napp.get('/health', async (req, res) => {\n const checks = {\n database: await checkDatabase(),\n redis: await checkRedis(),\n memory: checkMemory(),\n uptime: process.uptime()\n };\n\n res.json({\n status: 'healthy',\n version: process.env.VERSION,\n checks\n });\n});\n```\n\n**Load Balancer Health Check**:\n\n```nginx\n# nginx.conf\nupstream backend {\n server api1:8080;\n server api2:8080;\n server api3:8080;\n}\n\nserver {\n location / {\n proxy_pass http://backend;\n proxy_next_upstream error timeout http_503;\n proxy_connect_timeout 5s;\n }\n}\n```\n\n---\n\n### 6. Zero-Downtime Deployment Checklist\n\n```yaml\n# Complete production-ready compose\n\nservices:\n api:\n image: ${REGISTRY}/api:${VERSION}\n\n # Health check for readiness\n healthcheck:\n test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8080/health/ready\"]\n interval: 10s\n timeout: 5s\n retries: 3\n start_period: 30s\n\n # Graceful shutdown\n stop_grace_period: 30s\n\n # Rolling update config\n deploy:\n replicas: 3\n update_config:\n parallelism: 1\n delay: 10s\n order: start-first\n failure_action: rollback\n rollback_config:\n parallelism: 1\n delay: 5s\n resources:\n limits:\n memory: 512M\n\n # Connection draining\n labels:\n - \"traefik.http.services.api.loadbalancer.healthcheck.interval=10s\"\n - \"traefik.http.services.api.loadbalancer.healthcheck.timeout=5s\"\n\n networks:\n - frontend\n - backend\n```\n\n**Deployment Checklist Script**:\n\n```bash\n#!/bin/bash\n# deploy-checklist.sh\n\nVERSION=$1\nREGISTRY=${REGISTRY:-docker.io}\nIMAGE=$REGISTRY/myapp:$VERSION\n\necho \"=== Pre-Deployment Checklist ===\"\n\n# 1. Verify image exists\necho \"Checking image exists...\"\ndocker manifest inspect $IMAGE > /dev/null 2>&1 || {\n echo \"❌ Image not found: $IMAGE\"\n exit 1\n}\necho \"✅ Image verified\"\n\n# 2. Check current health\necho \"Checking current deployment health...\"\nUNHEALTHY=$(docker compose ps --format json | jq -r 'select(.Health != \"healthy\")' | wc -l)\nif [ \"$UNHEALTHY\" -gt 0 ]; then\n echo \"⚠️ Warning: $UNHEALTHY unhealthy containers\"\n read -p \"Continue anyway? (y/n) \" -n 1 -r\n echo\n if [[ ! $REPLY =~ ^[Yy]$ ]]; then\n exit 1\n fi\nfi\necho \"✅ Current deployment healthy\"\n\n# 3. Run migrations\necho \"Running migrations...\"\ndocker compose --profile migrate run --rm migrate || {\n echo \"❌ Migration failed\"\n exit 1\n}\necho \"✅ Migrations complete\"\n\n# 4. Deploy with rolling update\necho \"Starting rolling update...\"\nexport VERSION\ndocker compose up -d --no-deps api\n\n# 5. Wait for rollout\necho \"Waiting for rollout to complete...\"\nfor i in {1..60}; do\n READY=$(docker compose ps --format json | jq -r 'select(.Health == \"healthy\")' | wc -l)\n TOTAL=$(docker compose ps -q api | wc -l)\n\n if [ \"$READY\" -eq \"$TOTAL\" ] && [ \"$TOTAL\" -gt 0 ]; then\n echo \"✅ All containers healthy ($READY/$TOTAL)\"\n break\n fi\n\n echo \"Progress: $READY/$TOTAL healthy\"\n sleep 5\ndone\n\n# 6. Verify new version\necho \"Verifying deployed version...\"\nDEPLOYED=$(docker compose exec api cat /app/version.txt 2>/dev/null || echo \"unknown\")\nif [ \"$DEPLOYED\" != \"$VERSION\" ]; then\n echo \"⚠️ Warning: Version mismatch. Expected $VERSION, got $DEPLOYED\"\nfi\n\necho \"=== Deployment Complete ===\"\ndocker compose ps\n```\n\n---\n\n### 7. Rollback Strategies\n\n**Quick Rollback Script**:\n\n```bash\n#!/bin/bash\n# rollback.sh\n\nPREVIOUS_VERSION=$(cat .previous-version 2>/dev/null)\n\nif [ -z \"$PREVIOUS_VERSION\" ]; then\n echo \"No previous version found. Manual rollback required.\"\n echo \"Available versions:\"\n docker images myapp --format \"{{.Tag}}\" | head -10\n exit 1\nfi\n\necho \"Rolling back to version: $PREVIOUS_VERSION\"\n\nexport VERSION=$PREVIOUS_VERSION\ndocker compose up -d api\n\n# Swap version files\nmv .current-version .failed-version\nmv .previous-version .current-version\n\necho \"Rollback complete\"\n```\n\n**Automated Rollback on Failure**:\n\n```bash\n#!/bin/bash\n# deploy-with-auto-rollback.sh\n\nVERSION=$1\nPREVIOUS_VERSION=$(cat .current-version 2>/dev/null || echo \"unknown\")\n\nexport VERSION\n\n# Save current version\necho $PREVIOUS_VERSION > .previous-version\n\n# Deploy\ndocker compose up -d api\n\n# Wait and verify\nsleep 30\n\n# Check health\nHEALTHY=$(docker compose ps --format json | jq -r 'select(.Health == \"healthy\")' | wc -l)\nREQUIRED=3\n\nif [ \"$HEALTHY\" -lt \"$REQUIRED\" ]; then\n echo \"Deployment failed! Only $HEALTHY/$REQUIRED healthy. Rolling back...\"\n\n export VERSION=$PREVIOUS_VERSION\n docker compose up -d api\n\n echo \"Rolled back to $PREVIOUS_VERSION\"\n exit 1\nfi\n\n# Save new version as current\necho $VERSION > .current-version\necho \"Deployment successful\"\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Stop-First Update Order\n\n```yaml\n# ❌ Wrong: Stops old before starting new (downtime!)\ndeploy:\n update_config:\n order: stop-first\n\n# ✅ Right: Start new before stopping old\ndeploy:\n update_config:\n order: start-first\n```\n\n### Mistake 2: No Health Check in Deployment\n\n```yaml\n# ❌ Wrong: No way to know if new container is ready\nservices:\n api:\n image: myapp:latest\n # No healthcheck!\n\n# ✅ Right: Health check ensures readiness\nservices:\n api:\n image: myapp:latest\n healthcheck:\n test: curl -f http://localhost:8080/health/ready\n start_period: 30s\n```\n\n### Mistake 3: Breaking Database Changes\n\n```sql\n-- ❌ Wrong: Breaks v1 while deploying v2\nALTER TABLE users RENAME COLUMN name TO full_name;\n\n-- ✅ Right: Backwards compatible migration\n-- Step 1 (with v1 running):\nALTER TABLE users ADD COLUMN full_name VARCHAR(255);\nUPDATE users SET full_name = name;\n\n-- Step 2 (deploy v2 that uses full_name)\n-- Step 3 (later, when v1 is gone):\nALTER TABLE users DROP COLUMN name;\n```\n\n### Mistake 4: No Rollback Plan\n\n```bash\n# ❌ Wrong: Deploy and pray\ndocker compose up -d\n\n# ✅ Right: Always have rollback ready\necho $CURRENT_VERSION > .previous-version\ndocker compose up -d\n# Keep old image available for quick rollback\n```\n\n### Mistake 5: All-at-Once Updates\n\n```yaml\n# ❌ Wrong: Updates all containers simultaneously\ndeploy:\n update_config:\n parallelism: 0 # All at once\n\n# ✅ Right: One at a time with delay\ndeploy:\n update_config:\n parallelism: 1\n delay: 10s\n```\n\n---\n\n## 🐛 Debug This\n\nDeployment completes but application is down:\n\n```bash\n$ docker compose up -d api\n[+] Running 3/3\n ✔ Container api-1 Started\n ✔ Container api-2 Started\n ✔ Container api-3 Started\n\n$ curl http://localhost/health\ncurl: (7) Failed to connect\n```\n\nAll containers show as \"Started\" but the service is unreachable. What's happening?\n\n

Click to reveal analysis

\n\n**Investigation**:\n\n```bash\n# Check container health\n$ docker compose ps\nNAME STATUS PORTS\napi-1 Up 10s (health: starting) 8080/tcp\napi-2 Up 8s (health: starting) 8080/tcp\napi-3 Up 6s (health: starting) 8080/tcp\n\n# Check logs\n$ docker compose logs api\napi-1 | Starting server...\napi-1 | Connecting to database...\napi-1 | Error: Connection refused to db:5432\n```\n\n**Root Cause**: Database isn't ready, but `depends_on` wasn't using `service_healthy` condition.\n\n**Fix**:\n\n```yaml\nservices:\n api:\n depends_on:\n db:\n condition: service_healthy # Wait for DB to be ready\n\n db:\n healthcheck:\n test: pg_isready -U postgres\n interval: 5s\n timeout: 5s\n retries: 5\n```\n\n**Additional Fix**: Add retry logic in application for resilience:\n\n```javascript\nconst connectWithRetry = async (maxRetries = 10) => {\n for (let i = 0; i < maxRetries; i++) {\n try {\n await db.connect();\n return;\n } catch (err) {\n console.log(`DB connection attempt ${i + 1}/${maxRetries} failed`);\n await sleep(2000);\n }\n }\n throw new Error('Could not connect to database');\n};\n```\n

\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Rolling Update\nImplement a rolling update that:\n- Updates 1 container at a time\n- Waits for health check before continuing\n- Rolls back on failure\n\n### Exercise 2: Blue-Green Deployment\nSet up blue-green with:\n- Two parallel environments\n- Nginx switching between them\n- Instant rollback capability\n\n### Exercise 3: Canary Deployment\nImplement canary that:\n- Sends 10% of traffic to new version\n- Gradually increases to 100%\n- Can rollback instantly\n\n### Exercise 4: Database Migration\nHandle a schema change that:\n- Renames a column\n- Works during rolling update\n- Allows rollback\n\n### Exercise 5: Complete Pipeline\nCreate an end-to-end deployment:\n- Build and test in CI\n- Deploy to staging automatically\n- Deploy to production with approval\n- Automatic rollback on failure\n\n---\n\n## 🎤 Interview Questions\n\n### Q1: Explain the difference between rolling update, blue-green, and canary deployments.\n\n**Answer**:\n\n| Strategy | Traffic Pattern | Rollback Speed | Resource Cost |\n|----------|----------------|----------------|---------------|\n| Rolling | Gradual, container by container | Slow (redeploy) | Low |\n| Blue-Green | Instant switch | Instant | High (2x) |\n| Canary | Gradual % increase | Fast | Medium |\n\n**Rolling Update**:\n- Updates containers one at a time\n- Always some serving traffic\n- Rollback requires redeployment\n- Best for: Standard deployments\n\n**Blue-Green**:\n- Two complete environments\n- Traffic switches instantly\n- Instant rollback (switch back)\n- Best for: Critical systems, instant rollback needs\n\n**Canary**:\n- New version gets small traffic %\n- Monitor, then increase\n- Quick rollback (send 0% to canary)\n- Best for: Testing in production, risky changes\n\n---\n\n### Q2: How do you handle database migrations in a zero-downtime deployment?\n\n**Answer**: Use backwards-compatible migrations with expand-contract pattern:\n\n**Phase 1 - Expand** (add new, keep old):\n```sql\n-- Add new column without removing old\nALTER TABLE users ADD COLUMN full_name VARCHAR(255);\n-- Backfill\nUPDATE users SET full_name = name;\n-- Create trigger to keep in sync\nCREATE TRIGGER sync_names ...\n```\n\n**Phase 2 - Deploy** (use both):\n```javascript\n// v2 code writes to both columns\nuser.full_name = value;\nuser.name = value; // Keep old updated\n```\n\n**Phase 3 - Contract** (remove old):\n```sql\n-- Only after all instances are v2\nALTER TABLE users DROP COLUMN name;\n```\n\n**Key principles**:\n1. Never rename columns directly\n2. Never drop columns until all code is updated\n3. Always be able to roll back\n\n---\n\n### Q3: What's the `order: start-first` configuration and why is it important?\n\n**Answer**: `order: start-first` means Docker starts the new container BEFORE stopping the old one.\n\n**Without start-first** (stop-first):\n```\n1. Stop old container\n2. (Service unavailable!)\n3. Start new container\n4. Wait for health check\n5. Service available again\n```\n\n**With start-first**:\n```\n1. Start new container\n2. Wait for health check\n3. (Old still serving traffic)\n4. Stop old container\n5. Zero downtime!\n```\n\n```yaml\ndeploy:\n update_config:\n order: start-first # Critical for zero-downtime\n failure_action: rollback\n```\n\n**Requirement**: Need enough resources to run both old and new simultaneously. For a 3-replica service, briefly need capacity for 4.\n\n---\n\n### Q4: How do you implement automatic rollback on deployment failure?\n\n**Answer**: Combine health checks with failure actions:\n\n```yaml\nservices:\n api:\n healthcheck:\n test: curl -f http://localhost:8080/health/ready\n interval: 10s\n retries: 3\n start_period: 30s\n\n deploy:\n update_config:\n failure_action: rollback # Auto-rollback on health failure\n monitor: 60s # Watch for 60s after deploy\n rollback_config:\n parallelism: 1\n delay: 5s\n```\n\n**For Compose (non-Swarm)**:\n\n```bash\n#!/bin/bash\n# Deploy with timeout rollback\n\nTIMEOUT=120\nVERSION=$1\nPREV_VERSION=$(cat .version)\n\nexport VERSION\ndocker compose up -d api\n\n# Monitor health\nfor i in $(seq 1 $((TIMEOUT/10))); do\n HEALTHY=$(docker compose ps --format json | jq 'select(.Health==\"healthy\")' | wc -l)\n TOTAL=$(docker compose ps -q api | wc -l)\n\n if [ \"$HEALTHY\" -eq \"$TOTAL\" ]; then\n echo $VERSION > .version\n echo \"Deployment successful\"\n exit 0\n fi\n sleep 10\ndone\n\n# Timeout - rollback\necho \"Health check timeout. Rolling back...\"\nexport VERSION=$PREV_VERSION\ndocker compose up -d api\nexit 1\n```\n\n---\n\n### Q5: How do you handle a scenario where you need to deploy a breaking change?\n\n**Answer**: Use feature flags and gradual rollout:\n\n**Step 1**: Deploy code with feature flag OFF\n```javascript\nif (featureFlags.isEnabled('new-payment-system')) {\n useNewPaymentSystem();\n} else {\n useOldPaymentSystem();\n}\n```\n\n**Step 2**: Run database migrations\n```sql\n-- Add new tables/columns while old code still works\nCREATE TABLE new_payments (...);\n```\n\n**Step 3**: Enable feature for internal users\n```javascript\nfeatureFlags.enable('new-payment-system', { users: ['internal'] });\n```\n\n**Step 4**: Gradual rollout\n```javascript\nfeatureFlags.enable('new-payment-system', { percentage: 10 });\n// Monitor...\nfeatureFlags.enable('new-payment-system', { percentage: 50 });\n// Monitor...\nfeatureFlags.enable('new-payment-system', { percentage: 100 });\n```\n\n**Step 5**: Remove old code and flag (later release)\n```javascript\n// Remove feature flag check\nuseNewPaymentSystem();\n```\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Deployment Strategies\n\n| Strategy | Use When |\n|----------|----------|\n| Rolling | Standard deployments, most cases |\n| Blue-Green | Need instant rollback |\n| Canary | Testing risky changes |\n\n### Zero-Downtime Requirements\n1. `order: start-first` in deploy config\n2. Health checks with `start_period`\n3. Graceful shutdown handling\n4. Backwards-compatible migrations\n\n### Rollback Essentials\n- Always save previous version\n- Keep old images available\n- Automate rollback on failure\n- Test rollback procedure\n\n---\n\n## 📋 Quick Reference\n\n```yaml\n# Zero-downtime deploy config\ndeploy:\n update_config:\n order: start-first\n parallelism: 1\n delay: 10s\n failure_action: rollback\n\nhealthcheck:\n test: curl -f http://localhost:8080/health\n interval: 10s\n retries: 3\n start_period: 30s\n\nstop_grace_period: 30s\n```\n\n```bash\n# Quick rollback\nexport VERSION=$(cat .previous-version)\ndocker compose up -d api\n```\n\n---\n\n## 📅 Review Schedule\n\n- **Day 1**: Implement rolling update with health checks\n- **Day 3**: Set up blue-green deployment\n- **Day 7**: Practice database migrations\n- **Day 14**: Implement canary with monitoring\n- **Day 30**: Full deployment pipeline with rollback\n\n---\n\n## 📚 Series Navigation\n\n- **Previous**: [Part 17 - CI/CD Pipeline Patterns](/articles/docker-part-17-cicd)\n- **Next**: [Part 19 - Monitoring & Troubleshooting Production](/articles/docker-part-19-monitoring)\n- **Index**: [Docker Compendium Series](/series/docker-compendium)","2026-06-10T07:19:25.567077Z","2026-06-10T07:19:25.567664Z",18,{"_34":16,"_96":59,"_97":98,"_99":100,"_101":-5,"_102":-5,"_103":-5,"_104":-5,"_105":-5},{"_34":92,"_74":61,"_38":93,"_77":94,"_95":-5},[118,119,120],{"_34":16,"_74":75,"_38":76,"_77":78,"_79":24},{"_34":121,"_74":122,"_38":123},{"_34":86,"_74":87,"_38":88},{"_34":83,"_74":84,"_38":85},246,"Deployment","deployment",69,"Monitoring & Troubleshooting Production","docker-part-19-monitoring","Container metrics, resource analysis, post-mortem","## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| **Difficulty** | 🟠 Advanced |\n| **Prerequisites** | Part 11 (Logging), Part 13 (Debugging) |\n| **Stack** | Prometheus, Grafana, Loki, cAdvisor |\n| **Time Investment** | 28 minutes read + 90 minutes practice |\n| **Payoff** | Find issues before users report them |\n\n---\n\n## 🎯 What You'll Learn\n\nAfter this article, you'll be able to:\n\n1. **Set up metrics collection** with Prometheus and cAdvisor\n2. **Build dashboards** in Grafana for container monitoring\n3. **Implement centralized logging** with Loki\n4. **Create alerts** for production issues\n5. **Systematically troubleshoot** production problems\n\n---\n\n## 🔥 Production Story: The Silent Memory Leak\n\n**The Setup**: E-commerce site, holiday season. Everything running on Docker Compose. No monitoring \"because we're small.\"\n\n**Day 1**: Site runs fine.\n**Day 3**: Occasional slow responses.\n**Day 5**: 503 errors during peak hours.\n**Day 7**: Complete outage at 2 AM. Black Friday.\n\n**The Investigation** (too late):\n```bash\n$ dmesg | tail\n[timestamp] Out of memory: Kill process 1234 (java) score 900\n[timestamp] Out of memory: Kill process 1235 (node) score 850\n\n$ docker stats --no-stream\nNAME MEM USAGE / LIMIT\napi 1.9GB / 2GB # 95% memory!\nworker 1.8GB / 2GB # 90% memory!\n```\n\n**Root Cause**: Memory leak in API service. Without monitoring, nobody noticed the gradual increase until it was too late.\n\n**What Monitoring Would Have Shown**:\n```\nDay 1: api memory = 400MB ✓\nDay 3: api memory = 1.2GB ⚠️ Alert triggered\nDay 5: api memory = 1.6GB (restart, investigate)\n```\n\n**The Fix**: Prometheus + Grafana + alerts. Now they catch issues before users do.\n\n---\n\n## 🧠 Mental Model: Observability Stack\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ OBSERVABILITY PYRAMID │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ DASHBOARDS │ │\n│ │ (Grafana - Visualization) │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ ↑ │\n│ ┌──────────────┬────────┴────────┬──────────────┐ │\n│ │ METRICS │ LOGS │ TRACES │ │\n│ │ Prometheus │ Loki │ Jaeger │ │\n│ │ │ │ (optional) │ │\n│ └──────────────┴─────────────────┴──────────────┘ │\n│ ↑ │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ COLLECTORS │ │\n│ │ cAdvisor node_exporter Promtail App metrics │ │\n│ │ (containers) (host) (logs) (custom) │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ ↑ │\n│ ┌─────────────────────────────────────────────────────────┐ │\n│ │ DATA SOURCES │ │\n│ │ Docker Daemon Host OS Application Network │ │\n│ └─────────────────────────────────────────────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n\nData Flow:\n Containers → cAdvisor → Prometheus → Grafana → Alerts\n ↓\n Logs → Promtail → Loki → Grafana\n```\n\n---\n\n## 🔬 Deep Dive\n\n### 1. Metrics Collection with cAdvisor and Prometheus\n\n**Docker Compose Monitoring Stack**:\n\n```yaml\n# docker-compose.monitoring.yml\n\nservices:\n # ═══════════════════════════════════════════════════════════════\n # PROMETHEUS - Metrics database\n # ═══════════════════════════════════════════════════════════════\n prometheus:\n image: prom/prometheus:v2.47.0\n command:\n - '--config.file=/etc/prometheus/prometheus.yml'\n - '--storage.tsdb.path=/prometheus'\n - '--storage.tsdb.retention.time=15d'\n - '--web.enable-lifecycle'\n volumes:\n - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro\n - ./prometheus/alerts.yml:/etc/prometheus/alerts.yml:ro\n - prometheus_data:/prometheus\n ports:\n - \"9090:9090\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # GRAFANA - Visualization\n # ═══════════════════════════════════════════════════════════════\n grafana:\n image: grafana/grafana:10.1.0\n environment:\n - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}\n - GF_USERS_ALLOW_SIGN_UP=false\n volumes:\n - grafana_data:/var/lib/grafana\n - ./grafana/provisioning:/etc/grafana/provisioning:ro\n - ./grafana/dashboards:/var/lib/grafana/dashboards:ro\n ports:\n - \"3000:3000\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # cADVISOR - Container metrics\n # ═══════════════════════════════════════════════════════════════\n cadvisor:\n image: gcr.io/cadvisor/cadvisor:v0.47.2\n privileged: true\n volumes:\n - /:/rootfs:ro\n - /var/run:/var/run:ro\n - /sys:/sys:ro\n - /var/lib/docker/:/var/lib/docker:ro\n - /dev/disk/:/dev/disk:ro\n ports:\n - \"8080:8080\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # NODE EXPORTER - Host metrics\n # ═══════════════════════════════════════════════════════════════\n node-exporter:\n image: prom/node-exporter:v1.6.1\n command:\n - '--path.rootfs=/host'\n volumes:\n - /:/host:ro,rslave\n ports:\n - \"9100:9100\"\n networks:\n - monitoring\n restart: unless-stopped\n\n # ═══════════════════════════════════════════════════════════════\n # ALERTMANAGER - Alert routing\n # ═══════════════════════════════════════════════════════════════\n alertmanager:\n image: prom/alertmanager:v0.26.0\n volumes:\n - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro\n ports:\n - \"9093:9093\"\n networks:\n - monitoring\n restart: unless-stopped\n\nnetworks:\n monitoring:\n driver: bridge\n\nvolumes:\n prometheus_data:\n grafana_data:\n```\n\n**Prometheus Configuration**:\n\n```yaml\n# prometheus/prometheus.yml\n\nglobal:\n scrape_interval: 15s\n evaluation_interval: 15s\n\nalerting:\n alertmanagers:\n - static_configs:\n - targets: ['alertmanager:9093']\n\nrule_files:\n - /etc/prometheus/alerts.yml\n\nscrape_configs:\n # Prometheus itself\n - job_name: 'prometheus'\n static_configs:\n - targets: ['localhost:9090']\n\n # Container metrics from cAdvisor\n - job_name: 'cadvisor'\n static_configs:\n - targets: ['cadvisor:8080']\n\n # Host metrics from node-exporter\n - job_name: 'node'\n static_configs:\n - targets: ['node-exporter:9100']\n\n # Docker daemon metrics\n - job_name: 'docker'\n static_configs:\n - targets: ['host.docker.internal:9323']\n\n # Application metrics (if exposed)\n - job_name: 'app'\n static_configs:\n - targets: ['api:8080']\n metrics_path: /metrics\n```\n\n---\n\n### 2. Key Metrics to Monitor\n\n**Container Metrics** (from cAdvisor):\n\n```promql\n# CPU Usage (percentage of limit)\nsum(rate(container_cpu_usage_seconds_total{name=~\".+\"}[5m])) by (name) * 100\n\n# Memory Usage (bytes)\ncontainer_memory_usage_bytes{name=~\".+\"}\n\n# Memory Usage Percentage\ncontainer_memory_usage_bytes{name=~\".+\"} / container_spec_memory_limit_bytes{name=~\".+\"} * 100\n\n# Network I/O\nrate(container_network_receive_bytes_total{name=~\".+\"}[5m])\nrate(container_network_transmit_bytes_total{name=~\".+\"}[5m])\n\n# Disk I/O\nrate(container_fs_reads_bytes_total{name=~\".+\"}[5m])\nrate(container_fs_writes_bytes_total{name=~\".+\"}[5m])\n\n# Container restarts\nincrease(container_start_time_seconds{name=~\".+\"}[1h])\n```\n\n**Host Metrics** (from node-exporter):\n\n```promql\n# CPU Usage\n100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)\n\n# Memory Usage\n(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100\n\n# Disk Usage\n100 - ((node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"}) * 100)\n\n# Network\nrate(node_network_receive_bytes_total[5m])\nrate(node_network_transmit_bytes_total[5m])\n```\n\n---\n\n### 3. Alert Rules\n\n```yaml\n# prometheus/alerts.yml\n\ngroups:\n - name: container_alerts\n rules:\n # Container down\n - alert: ContainerDown\n expr: absent(container_last_seen{name=~\".+\"}) == 1\n for: 1m\n labels:\n severity: critical\n annotations:\n summary: \"Container {{ $labels.name }} is down\"\n description: \"Container has been down for more than 1 minute\"\n\n # High memory usage\n - alert: ContainerHighMemory\n expr: |\n (container_memory_usage_bytes{name=~\".+\"} /\n container_spec_memory_limit_bytes{name=~\".+\"}) > 0.85\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"Container {{ $labels.name }} memory usage > 85%\"\n description: \"Memory usage is {{ $value | printf \\\"%.1f\\\" }}%\"\n\n # Memory near limit (critical)\n - alert: ContainerMemoryCritical\n expr: |\n (container_memory_usage_bytes{name=~\".+\"} /\n container_spec_memory_limit_bytes{name=~\".+\"}) > 0.95\n for: 1m\n labels:\n severity: critical\n annotations:\n summary: \"Container {{ $labels.name }} memory critical\"\n description: \"Memory at {{ $value | printf \\\"%.1f\\\" }}%, OOM likely\"\n\n # High CPU usage\n - alert: ContainerHighCPU\n expr: |\n sum(rate(container_cpu_usage_seconds_total{name=~\".+\"}[5m])) by (name) > 0.8\n for: 10m\n labels:\n severity: warning\n annotations:\n summary: \"Container {{ $labels.name }} high CPU usage\"\n description: \"CPU usage is {{ $value | printf \\\"%.1f\\\" }}%\"\n\n # Container restarts\n - alert: ContainerRestarting\n expr: increase(container_start_time_seconds{name=~\".+\"}[1h]) > 3\n labels:\n severity: warning\n annotations:\n summary: \"Container {{ $labels.name }} restarting frequently\"\n description: \"{{ $value }} restarts in the last hour\"\n\n - name: host_alerts\n rules:\n # High host memory\n - alert: HostHighMemory\n expr: |\n (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) /\n node_memory_MemTotal_bytes > 0.9\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"Host memory usage > 90%\"\n\n # Disk space low\n - alert: HostDiskSpaceLow\n expr: |\n (node_filesystem_avail_bytes{mountpoint=\"/\"} /\n node_filesystem_size_bytes{mountpoint=\"/\"}) < 0.1\n for: 5m\n labels:\n severity: critical\n annotations:\n summary: \"Disk space < 10% remaining\"\n```\n\n**Alertmanager Configuration**:\n\n```yaml\n# alertmanager/alertmanager.yml\n\nglobal:\n resolve_timeout: 5m\n\nroute:\n group_by: ['alertname']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 1h\n receiver: 'slack-notifications'\n\n routes:\n - match:\n severity: critical\n receiver: 'pagerduty-critical'\n - match:\n severity: warning\n receiver: 'slack-notifications'\n\nreceivers:\n - name: 'slack-notifications'\n slack_configs:\n - api_url: '${SLACK_WEBHOOK_URL}'\n channel: '#alerts'\n title: '{{ .Status | toUpper }}: {{ .CommonAnnotations.summary }}'\n text: '{{ .CommonAnnotations.description }}'\n\n - name: 'pagerduty-critical'\n pagerduty_configs:\n - service_key: '${PAGERDUTY_KEY}'\n severity: critical\n```\n\n---\n\n### 4. Centralized Logging with Loki\n\n**Add Loki to Monitoring Stack**:\n\n```yaml\n# Add to docker-compose.monitoring.yml\n\nservices:\n loki:\n image: grafana/loki:2.9.0\n command: -config.file=/etc/loki/local-config.yaml\n volumes:\n - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro\n - loki_data:/loki\n ports:\n - \"3100:3100\"\n networks:\n - monitoring\n restart: unless-stopped\n\n promtail:\n image: grafana/promtail:2.9.0\n volumes:\n - ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro\n - /var/lib/docker/containers:/var/lib/docker/containers:ro\n - /var/run/docker.sock:/var/run/docker.sock\n command: -config.file=/etc/promtail/config.yml\n networks:\n - monitoring\n restart: unless-stopped\n\nvolumes:\n loki_data:\n```\n\n**Promtail Configuration**:\n\n```yaml\n# promtail/promtail-config.yml\n\nserver:\n http_listen_port: 9080\n grpc_listen_port: 0\n\npositions:\n filename: /tmp/positions.yaml\n\nclients:\n - url: http://loki:3100/loki/api/v1/push\n\nscrape_configs:\n - job_name: docker\n docker_sd_configs:\n - host: unix:///var/run/docker.sock\n refresh_interval: 5s\n relabel_configs:\n # Use container name as label\n - source_labels: ['__meta_docker_container_name']\n regex: '/(.*)'\n target_label: 'container'\n # Add compose service label\n - source_labels: ['__meta_docker_container_label_com_docker_compose_service']\n target_label: 'service'\n # Add compose project label\n - source_labels: ['__meta_docker_container_label_com_docker_compose_project']\n target_label: 'project'\n```\n\n**Loki Configuration**:\n\n```yaml\n# loki/loki-config.yml\n\nauth_enabled: false\n\nserver:\n http_listen_port: 3100\n\ningester:\n lifecycler:\n ring:\n kvstore:\n store: inmemory\n replication_factor: 1\n chunk_idle_period: 5m\n chunk_retain_period: 30s\n\nschema_config:\n configs:\n - from: 2020-01-01\n store: boltdb-shipper\n object_store: filesystem\n schema: v11\n index:\n prefix: index_\n period: 24h\n\nstorage_config:\n boltdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/cache\n shared_store: filesystem\n filesystem:\n directory: /loki/chunks\n\nlimits_config:\n enforce_metric_name: false\n reject_old_samples: true\n reject_old_samples_max_age: 168h\n\nchunk_store_config:\n max_look_back_period: 0s\n\ntable_manager:\n retention_deletes_enabled: true\n retention_period: 168h # 7 days\n```\n\n**LogQL Queries in Grafana**:\n\n```logql\n# All logs from a container\n{container=\"api\"}\n\n# Error logs only\n{container=\"api\"} |= \"error\"\n\n# JSON log parsing\n{container=\"api\"} | json | level=\"error\"\n\n# Count errors per minute\ncount_over_time({container=\"api\"} |= \"error\" [1m])\n\n# Top error messages\n{container=\"api\"} | json | level=\"error\"\n | line_format \"{{ .message }}\"\n | topk(10, count_over_time({container=\"api\"} [1h]) by (message))\n\n# Response time from logs (if logged)\n{container=\"api\"} | json | unwrap duration [5m]\n | quantile_over_time(0.95, 5m)\n```\n\n---\n\n### 5. Grafana Dashboard for Docker\n\n**Dashboard JSON** (simplified):\n\n```json\n{\n \"title\": \"Docker Container Monitoring\",\n \"panels\": [\n {\n \"title\": \"Container CPU Usage\",\n \"type\": \"timeseries\",\n \"targets\": [\n {\n \"expr\": \"sum(rate(container_cpu_usage_seconds_total{name=~\\\".+\\\"}[5m])) by (name) * 100\",\n \"legendFormat\": \"{{name}}\"\n }\n ]\n },\n {\n \"title\": \"Container Memory Usage\",\n \"type\": \"timeseries\",\n \"targets\": [\n {\n \"expr\": \"container_memory_usage_bytes{name=~\\\".+\\\"} / 1024 / 1024\",\n \"legendFormat\": \"{{name}}\"\n }\n ]\n },\n {\n \"title\": \"Container Memory % of Limit\",\n \"type\": \"gauge\",\n \"targets\": [\n {\n \"expr\": \"(container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100\",\n \"legendFormat\": \"{{name}}\"\n }\n ],\n \"thresholds\": [\n { \"value\": 0, \"color\": \"green\" },\n { \"value\": 70, \"color\": \"yellow\" },\n { \"value\": 85, \"color\": \"red\" }\n ]\n },\n {\n \"title\": \"Container Network I/O\",\n \"type\": \"timeseries\",\n \"targets\": [\n {\n \"expr\": \"rate(container_network_receive_bytes_total{name=~\\\".+\\\"}[5m])\",\n \"legendFormat\": \"{{name}} RX\"\n },\n {\n \"expr\": \"rate(container_network_transmit_bytes_total{name=~\\\".+\\\"}[5m])\",\n \"legendFormat\": \"{{name}} TX\"\n }\n ]\n },\n {\n \"title\": \"Container Restarts (1h)\",\n \"type\": \"stat\",\n \"targets\": [\n {\n \"expr\": \"increase(container_start_time_seconds{name=~\\\".+\\\"}[1h])\",\n \"legendFormat\": \"{{name}}\"\n }\n ]\n }\n ]\n}\n```\n\n**Grafana Provisioning**:\n\n```yaml\n# grafana/provisioning/datasources/datasources.yml\n\napiVersion: 1\n\ndatasources:\n - name: Prometheus\n type: prometheus\n access: proxy\n url: http://prometheus:9090\n isDefault: true\n\n - name: Loki\n type: loki\n access: proxy\n url: http://loki:3100\n```\n\n---\n\n### 6. Troubleshooting Runbook\n\n**Systematic Troubleshooting Process**:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ TROUBLESHOOTING FLOWCHART │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ SYMPTOM DETECTED │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Check Dashboard │ → Memory, CPU, Network, Errors │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Check Logs │ → Recent errors, patterns │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Check Resources │ → docker stats, df -h, free │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Deep Dive │ → exec, inspect, strace │\n│ └────────┬────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────┐ │\n│ │ Take Action │ → Restart, Scale, Rollback │\n│ └─────────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**Quick Diagnosis Script**:\n\n```bash\n#!/bin/bash\n# diagnose.sh - Quick production diagnosis\n\necho \"═══════════════════════════════════════════════════════════════\"\necho \"DOCKER PRODUCTION DIAGNOSIS\"\necho \"Time: $(date)\"\necho \"═══════════════════════════════════════════════════════════════\"\n\necho -e \"\\n### Container Status ###\"\ndocker compose ps\n\necho -e \"\\n### Container Resources ###\"\ndocker stats --no-stream --format \"table {{.Name}}\\t{{.CPUPerc}}\\t{{.MemUsage}}\\t{{.MemPerc}}\"\n\necho -e \"\\n### Unhealthy Containers ###\"\ndocker ps --filter health=unhealthy --format \"{{.Names}}: {{.Status}}\"\n\necho -e \"\\n### Recent Restarts ###\"\ndocker ps --filter \"status=restarting\" --format \"{{.Names}}: {{.Status}}\"\n\necho -e \"\\n### Disk Usage ###\"\ndf -h / /var/lib/docker\n\necho -e \"\\n### Docker Disk Usage ###\"\ndocker system df\n\necho -e \"\\n### Recent Docker Events ###\"\ndocker events --since \"10m\" --until \"0s\" 2>/dev/null | tail -20\n\necho -e \"\\n### Recent Error Logs (all containers) ###\"\ndocker compose logs --since \"10m\" 2>&1 | grep -i -E \"(error|exception|fatal)\" | tail -20\n\necho -e \"\\n### Host Memory ###\"\nfree -h\n\necho -e \"\\n### Host Load ###\"\nuptime\n```\n\n---\n\n### 7. Common Issues and Solutions\n\n**Issue: Container OOM Killed**\n\n```bash\n# Detect\ndocker inspect container --format='{{.State.OOMKilled}}'\ndmesg | grep -i \"killed process\"\n\n# Diagnose\ndocker stats container --no-stream\ndocker exec container cat /sys/fs/cgroup/memory.current\ndocker exec container cat /sys/fs/cgroup/memory.max\n\n# Fix options\n1. Increase memory limit in compose\n2. Fix memory leak in application\n3. Optimize application memory usage (JVM: -XX:MaxRAMPercentage)\n```\n\n**Issue: Container Restarting Loop**\n\n```bash\n# Detect\ndocker ps # Shows \"Restarting (1) 5 seconds ago\"\ndocker inspect container --format='{{.RestartCount}}'\n\n# Diagnose\ndocker logs container --tail 100\ndocker inspect container --format='{{.State.ExitCode}}'\n# Common exit codes:\n# 137 = OOM or SIGKILL\n# 1 = Application error\n# 126 = Permission denied\n# 127 = Command not found\n\n# Fix\ndocker compose stop container\ndocker logs container # Check logs\n# Fix the issue, then:\ndocker compose up -d container\n```\n\n**Issue: High CPU Usage**\n\n```bash\n# Detect\ndocker stats --no-stream | sort -k3 -h -r | head -5\n\n# Diagnose\n# Get PID in container\ndocker exec container top -b -n 1\n\n# Profile\ndocker exec container sh -c \"\n while true; do\n date >> /tmp/profile.txt\n cat /proc/loadavg >> /tmp/profile.txt\n ps aux >> /tmp/profile.txt\n sleep 5\n done\n\" &\n\n# Fix options\n1. Scale horizontally (more containers)\n2. Add CPU limits to prevent hogging\n3. Profile application code\n```\n\n**Issue: Network Connectivity**\n\n```bash\n# Detect\ndocker exec container curl -v http://other-service:8080\n\n# Diagnose\n# Check DNS\ndocker exec container nslookup other-service\n\n# Check network\ndocker network ls\ndocker network inspect network_name\n\n# Check from another container\ndocker run --rm --network=mynetwork nicolaka/netshoot \\\n curl -v http://other-service:8080\n\n# Fix options\n1. Ensure containers on same network\n2. Check service name spelling\n3. Check if target service is healthy\n```\n\n---\n\n### 8. Production Monitoring Checklist\n\n```yaml\n# monitoring-checklist.yml\n\n# Essential Metrics to Monitor\nmetrics:\n container:\n - memory_usage_percent # Alert at 85%\n - cpu_usage_percent # Alert at 80%\n - restart_count # Alert at > 3/hour\n - health_status # Alert on unhealthy\n\n host:\n - disk_usage_percent # Alert at 90%\n - memory_usage_percent # Alert at 90%\n - cpu_usage_percent # Alert at 90%\n - network_errors # Alert on increase\n\n application:\n - request_latency_p99 # Alert at > 1s\n - error_rate # Alert at > 1%\n - request_rate # Alert on sudden drop\n\n# Essential Logs to Watch\nlogs:\n - level: error # All errors\n - level: warn # Warnings\n - message: \"OOM\" # Memory issues\n - message: \"connection refused\" # Network issues\n - message: \"timeout\" # Timeout issues\n\n# Dashboard Panels\ndashboards:\n overview:\n - Container status grid\n - CPU usage by container\n - Memory usage by container\n - Network I/O\n - Recent errors\n\n per_service:\n - Request rate\n - Error rate\n - Latency percentiles\n - Resource usage\n - Logs panel\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: No Alerts Until Outage\n\n```yaml\n# ❌ Wrong: Just dashboards, no alerts\n# Nobody watches dashboards at 3 AM\n\n# ✅ Right: Alerts for key metrics\n- alert: ContainerHighMemory\n expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85\n for: 5m\n```\n\n### Mistake 2: Alert Fatigue\n\n```yaml\n# ❌ Wrong: Alert on everything\n- alert: MinorWarning\n expr: cpu > 50% # Too sensitive\n for: 1m # Too quick\n\n# ✅ Right: Meaningful thresholds\n- alert: HighCPU\n expr: cpu > 85% # Actually concerning\n for: 10m # Sustained issue\n```\n\n### Mistake 3: No Log Retention Policy\n\n```yaml\n# ❌ Wrong: Keep logs forever\n# Disk fills up, system crashes\n\n# ✅ Right: Set retention limits\nservices:\n app:\n logging:\n options:\n max-size: \"10m\"\n max-file: \"3\"\n\n# And in Loki\nlimits_config:\n retention_period: 168h # 7 days\n```\n\n### Mistake 4: Missing Context in Logs\n\n```javascript\n// ❌ Wrong: Generic errors\nconsole.error(\"Error processing request\");\n\n// ✅ Right: Context-rich logs\nlogger.error({\n message: \"Error processing request\",\n requestId: req.id,\n userId: req.user?.id,\n path: req.path,\n error: err.message,\n stack: err.stack\n});\n```\n\n### Mistake 5: No Baseline\n\n```yaml\n# ❌ Wrong: Alert on absolute value\n- alert: HighMemory\n expr: container_memory_usage_bytes > 1000000000 # 1GB\n # What if the limit is 2GB? Or 500MB?\n\n# ✅ Right: Alert on percentage of limit\n- alert: HighMemory\n expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85\n```\n\n---\n\n## 🐛 Debug This\n\nProduction alert: \"Container api memory > 90%\". Dashboard shows memory climbing steadily for the past 3 days.\n\n```\nDay 1: 40% → Day 2: 60% → Day 3: 80% → Now: 92%\n```\n\nNo code changes were deployed. What's happening and how do you fix it?\n\n

Click to reveal analysis

\n\n**Diagnosis**:\nThis is a classic memory leak pattern - steady growth without code changes means:\n1. Application-level memory leak\n2. Connection leak (database, redis, etc.)\n3. Cache without eviction\n4. Event listeners not cleaned up\n\n**Investigation steps**:\n\n```bash\n# 1. Get memory breakdown\ndocker exec api node --expose-gc -e \"\n const v8 = require('v8');\n console.log(v8.getHeapStatistics());\n\"\n\n# 2. Check connection counts\ndocker exec api netstat -an | grep ESTABLISHED | wc -l\n\n# 3. Check database connections\ndocker exec api sh -c \"\n curl http://db:5432/pools # If using pgBouncer\n\"\n\n# 4. Generate heap snapshot (Node.js)\ndocker exec api kill -USR2 1 # If app supports it\n\n# 5. Check for patterns\ndocker compose logs api --since \"72h\" | grep -i \"leak\\|memory\\|heap\"\n```\n\n**Common fixes**:\n\n1. **Restart immediately** to restore service:\n ```bash\n docker compose restart api\n ```\n\n2. **Investigate root cause**:\n - Add memory profiling to application\n - Check for unclosed database connections\n - Look for growing arrays/objects\n - Check event listener cleanup\n\n3. **Add protection**:\n ```yaml\n # Compose: auto-restart on high memory\n healthcheck:\n test: |\n MEMORY=$(cat /sys/fs/cgroup/memory.current)\n LIMIT=$(cat /sys/fs/cgroup/memory.max)\n if [ $((MEMORY * 100 / LIMIT)) -gt 90 ]; then exit 1; fi\n curl -f http://localhost:8080/health\n ```\n\n4. **Long-term fix**: Profile application, find and fix the leak.\n

\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Basic Monitoring Stack\nSet up Prometheus + Grafana + cAdvisor for a simple web application. Create a dashboard showing:\n- Container CPU usage\n- Container memory usage\n- Network I/O\n\n### Exercise 2: Alerting\nAdd alertmanager and configure alerts for:\n- Container memory > 85%\n- Container restarting > 3 times/hour\n- Host disk space < 10%\n\n### Exercise 3: Centralized Logging\nAdd Loki and Promtail. Create LogQL queries for:\n- All errors from a specific service\n- Request latency > 1s\n- Count of errors per minute\n\n### Exercise 4: Custom Metrics\nAdd application metrics (request count, latency histogram) and:\n- Expose them on /metrics endpoint\n- Scrape with Prometheus\n- Create dashboard panel\n\n### Exercise 5: Runbook\nCreate a troubleshooting runbook for:\n- High memory usage\n- Service unreachable\n- High error rate\nInclude detection, diagnosis, and remediation steps.\n\n---\n\n## 🎤 Interview Questions\n\n### Q1: How do you monitor Docker containers in production?\n\n**Answer**: A complete monitoring stack includes:\n\n1. **Metrics Collection**:\n - cAdvisor for container metrics (CPU, memory, network, disk)\n - node-exporter for host metrics\n - Application metrics exposed on /metrics\n\n2. **Storage and Querying**:\n - Prometheus for metrics\n - Loki for logs\n - Retention policies for both\n\n3. **Visualization**:\n - Grafana dashboards\n - Key panels: resource usage, error rates, latency\n\n4. **Alerting**:\n - Prometheus alerting rules\n - Alertmanager for routing\n - Slack/PagerDuty integration\n\nKey metrics to watch:\n- Memory usage % of limit (alert at 85%)\n- Container restart count\n- Error rate and latency\n- Disk space\n\n---\n\n### Q2: How do you troubleshoot a container that's using high CPU?\n\n**Answer**: Systematic approach:\n\n1. **Identify** the container:\n ```bash\n docker stats --no-stream | sort -k3 -h -r\n ```\n\n2. **Check** what process is consuming CPU:\n ```bash\n docker exec container top -b -n 1\n docker exec container ps aux --sort=-%cpu\n ```\n\n3. **Analyze** the application:\n ```bash\n # For Node.js\n docker exec container node --prof-process\n # For Java\n docker exec container jstack PID\n ```\n\n4. **Check** for patterns:\n - Spike vs sustained high CPU\n - Correlation with traffic\n - Recent changes\n\n5. **Take action**:\n - Scale horizontally if legitimate load\n - Add CPU limits to prevent hogging\n - Fix code if inefficiency found\n - Rollback if recent deployment caused it\n\n---\n\n### Q3: What's the difference between container memory usage and the memory limit?\n\n**Answer**:\n\n**Memory Usage** (`container_memory_usage_bytes`):\n- Current memory used by the container\n- Includes RSS, cache, swap\n\n**Memory Limit** (`container_spec_memory_limit_bytes`):\n- Maximum allowed by cgroup\n- Set via `--memory` or compose `deploy.resources.limits.memory`\n\n**Why this matters**:\n\n```promql\n# Percentage is more meaningful than absolute value\ncontainer_memory_usage_bytes / container_spec_memory_limit_bytes * 100\n```\n\nWhen usage approaches limit:\n- 70-85%: Warning zone\n- 85-95%: Critical, action needed\n- >95%: OOM kill imminent\n\n**Best practice**: Alert on percentage, not absolute value. A container using 500MB might be fine (if limit is 2GB) or critical (if limit is 512MB).\n\n---\n\n### Q4: How do you implement log aggregation for Docker containers?\n\n**Answer**: Using Loki stack:\n\n1. **Deploy Loki** (log database):\n ```yaml\n services:\n loki:\n image: grafana/loki:2.9.0\n ports:\n - \"3100:3100\"\n ```\n\n2. **Deploy Promtail** (log collector):\n ```yaml\n promtail:\n image: grafana/promtail:2.9.0\n volumes:\n - /var/lib/docker/containers:/var/lib/docker/containers:ro\n - /var/run/docker.sock:/var/run/docker.sock\n ```\n\n3. **Configure discovery**:\n ```yaml\n scrape_configs:\n - job_name: docker\n docker_sd_configs:\n - host: unix:///var/run/docker.sock\n ```\n\n4. **Query in Grafana**:\n ```logql\n {container=\"api\"} |= \"error\"\n {service=\"api\"} | json | level=\"error\"\n ```\n\n**Key considerations**:\n- Set retention limits (default 7 days)\n- Index only necessary labels\n- Use JSON logging for structured queries\n\n---\n\n### Q5: How do you prevent alert fatigue while still catching real issues?\n\n**Answer**: Strategic alert configuration:\n\n1. **Meaningful thresholds**:\n ```yaml\n # Bad: Alerts on minor fluctuations\n expr: cpu > 50%\n\n # Good: Alerts on sustained issues\n expr: cpu > 85%\n for: 10m\n ```\n\n2. **Appropriate severity levels**:\n - Critical: Pages on-call immediately\n - Warning: Slack notification, review next day\n - Info: Dashboard only\n\n3. **Alert on symptoms, not causes**:\n ```yaml\n # Better: User-facing impact\n - alert: HighErrorRate\n expr: rate(http_errors[5m]) > 0.01\n\n # Worse: Internal metric\n - alert: HighThreadCount\n expr: threads > 100\n ```\n\n4. **Group related alerts**:\n ```yaml\n route:\n group_by: ['service', 'severity']\n group_wait: 30s # Wait to group related alerts\n ```\n\n5. **Regular review**: If an alert never fires, consider removing. If it fires daily and is ignored, fix threshold.\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Monitoring Stack\n- **Metrics**: Prometheus + cAdvisor\n- **Logs**: Loki + Promtail\n- **Visualization**: Grafana\n- **Alerts**: Alertmanager\n\n### Key Metrics\n| Metric | Warning | Critical |\n|--------|---------|----------|\n| Memory % | 85% | 95% |\n| CPU % | 80% | 90% |\n| Disk % | 80% | 90% |\n| Restart count | 3/hour | 5/hour |\n\n### Troubleshooting Flow\n1. Check dashboards\n2. Check logs\n3. Check resources\n4. Deep dive (exec, inspect)\n5. Take action (restart, scale, rollback)\n\n---\n\n## 📋 Quick Reference\n\n```bash\n# Quick diagnosis\ndocker stats --no-stream\ndocker ps --filter health=unhealthy\ndocker compose logs --since 10m | grep -i error\n\n# Memory check\ndocker inspect $C --format='{{.State.OOMKilled}}'\n\n# CPU check\ndocker exec $C top -b -n 1\n\n# Network check\ndocker exec $C nslookup service-name\n```\n\n```promql\n# Key PromQL queries\ncontainer_memory_usage_bytes / container_spec_memory_limit_bytes * 100\nrate(container_cpu_usage_seconds_total[5m]) * 100\nincrease(container_start_time_seconds[1h])\n```\n\n---\n\n## 📅 Review Schedule\n\n- **Day 1**: Set up Prometheus + Grafana + cAdvisor\n- **Day 3**: Create basic dashboard\n- **Day 7**: Add alerting rules\n- **Day 14**: Add centralized logging\n- **Day 30**: Build complete runbook\n\n---\n\n## 📚 Series Navigation\n\n- **Previous**: [Part 18 - Production Deployment Patterns](/articles/docker-part-18-deployment)\n- **Next**: [Part 20 - Docker Cheatsheet & Decision Guide](/articles/docker-part-20-cheatsheet)\n- **Index**: [Docker Compendium Series](/series/docker-compendium)","2026-06-10T07:19:25.577113Z","2026-06-10T07:19:25.577756Z",19,{"_34":16,"_96":59,"_97":98,"_99":100,"_101":-5,"_102":-5,"_103":-5,"_104":-5,"_105":-5},{"_34":92,"_74":61,"_38":93,"_77":94,"_95":-5},[136,137,138],{"_34":16,"_74":75,"_38":76,"_77":78,"_79":24},{"_34":86,"_74":87,"_38":88},{"_34":139,"_74":140,"_38":141},{"_34":83,"_74":84,"_38":85},238,"Monitoring","monitoring"]