[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6},"actionData","errors","routes/blog-tag",{"_7":8,"_9":10},"tag",{"_33":92,"_73":93,"_37":94},"articles",{"_11":12,"_13":14,"_15":16,"_17":18,"_19":20,"_21":22,"_23":24,"_25":24,"_26":16,"_27":28},"content",[31,32],"pageable",{"_29":22,"_30":20},"totalElements",2,"totalPages",1,"size",10,"number",0,"first",true,"last","numberOfElements","empty",false,"page_number","page_size",{"_33":109,"_35":110,"_37":111,"_39":112,"_11":113,"_42":-5,"_43":24,"_44":28,"_45":46,"_47":114,"_49":115,"_51":52,"_53":116,"_55":116,"_56":57,"_58":59,"_60":61,"_62":116,"_63":57,"_64":117,"_65":118,"_67":119,"_69":120,"_71":121},{"_33":34,"_35":36,"_37":38,"_39":40,"_11":41,"_42":-5,"_43":24,"_44":28,"_45":46,"_47":48,"_49":50,"_51":52,"_53":54,"_55":54,"_56":57,"_58":59,"_60":61,"_62":54,"_63":57,"_64":18,"_65":66,"_67":68,"_69":70,"_71":72},"id",51,"title","Container Internals: What's Really Running","slug","docker-part-1-container-internals","excerpt","Understand namespaces, cgroups, and what happens during docker run","# Docker Compendium Part 1: Container Internals - What's Really Running\n\nYou run `docker run nginx` and something happens. But what exactly? This article pulls back the curtain on containers - they're not VMs, not magic, just Linux processes with clever isolation. Understanding this changes how you debug, optimize, and secure containers.\n\n## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| Topic | Linux namespaces, cgroups, container runtime |\n| Complexity | Advanced |\n| Prerequisites | Basic Linux process model, basic Docker usage |\n| Key Insight | Containers are processes with isolation, not lightweight VMs |\n| Time to Master | 3-4 hours |\n\n---\n\n## 🎯 What You'll Learn\n\n- **Linux Namespaces** - how Docker isolates processes, networks, and filesystems\n- **Control Groups (cgroups)** - how resource limits actually work\n- **The container lifecycle** - what happens from `docker run` to process exit\n- **OCI Runtime** - the actual binary that creates containers\n- **Why this matters** - debugging and optimization implications\n\n---\n\n## 🔥 Production Story: The Zombie Process Apocalypse\n\nA team deployed a Python application processing video files. Each request spawned FFmpeg subprocesses. After a week in production, the container showed 1,247 processes - most were zombies (defunct processes).\n\n**The symptoms:**\n- Container memory slowly grew\n- Process table filled up\n- Eventually: \"fork: Resource temporarily unavailable\"\n\n**Root cause:** The Python app wasn't reaping child processes. In a VM or on bare metal, the init system (PID 1) handles orphaned children. Inside a container, the app IS PID 1 - and Python's default signal handling doesn't reap children.\n\n**The fix:** One flag: `docker run --init nginx`\n\nThis injects a minimal init process (tini) as PID 1 that properly reaps zombies. Without understanding PID namespaces and init responsibilities, this bug was \"unexplainable.\"\n\n**The lesson:** Containers aren't VMs. The rules are different. PID 1 has special responsibilities, and your app might not be ready for them.\n\n---\n\n## 🧠 Mental Model: Containers = Processes + Isolation\n\nForget \"lightweight VMs.\" A container is:\n\n```\n┌────────────────────────────────────────────────────────────────────┐\n│ HOST KERNEL │\n│ (shared by all containers) │\n├─────────────────────────────────────────────────────────────────────┤\n│ │\n│ ┌──────────────────────┐ ┌──────────────────────┐ │\n│ │ Container A │ │ Container B │ │\n│ │ │ │ │ │\n│ │ ┌────────────────┐ │ │ ┌────────────────┐ │ │\n│ │ │ NAMESPACES │ │ │ │ NAMESPACES │ │ Isolation │\n│ │ │ ┌──────────┐ │ │ │ │ ┌──────────┐ │ │ │\n│ │ │ │ PID: 1 │ │ │ │ │ │ PID: 1 │ │ │ │\n│ │ │ │ NET: own │ │ │ │ │ │ NET: own │ │ │ │\n│ │ │ │ MNT: own │ │ │ │ │ │ MNT: own │ │ │ │\n│ │ │ │ UTS: own │ │ │ │ │ │ UTS: own │ │ │ │\n│ │ │ └──────────┘ │ │ │ │ └──────────┘ │ │ │\n│ │ └────────────────┘ │ │ └────────────────┘ │ │\n│ │ │ │ │ │\n│ │ ┌────────────────┐ │ │ ┌────────────────┐ │ │\n│ │ │ CGROUPS │ │ │ │ CGROUPS │ │ Resource │\n│ │ │ mem: 512MB │ │ │ │ mem: 1GB │ │ Limits │\n│ │ │ cpu: 0.5 │ │ │ │ cpu: 2.0 │ │ │\n│ │ └────────────────┘ │ │ └────────────────┘ │ │\n│ │ │ │ │ │\n│ │ ┌────────────────┐ │ │ ┌────────────────┐ │ │\n│ │ │ FILESYSTEM │ │ │ │ FILESYSTEM │ │ Overlay │\n│ │ │ (overlay) │ │ │ │ (overlay) │ │ FS │\n│ │ └────────────────┘ │ │ └────────────────┘ │ │\n│ └──────────────────────┘ └──────────────────────┘ │\n│ │\n│ Host PID 1234 ─────────────────── Host PID 5678 │\n│ (nginx in Container A) (java in Container B) │\n│ │\n└─────────────────────────────────────────────────────────────────────┘\n```\n\n**Key insight:** Container A's \"PID 1\" is actually PID 1234 on the host. The kernel is shared. Only the view is isolated.\n\n**VM vs Container:**\n\n| Aspect | Virtual Machine | Container |\n|--------|-----------------|-----------|\n| Isolation | Hardware level | OS level |\n| Kernel | Own kernel | Shared with host |\n| Boot time | Minutes | Milliseconds |\n| Memory overhead | GBs | MBs |\n| Process visibility | None | Host sees all |\n\n---\n\n## 🔬 Deep Dive\n\n### Linux Namespaces: The Isolation Mechanism\n\nNamespaces give each container its own view of system resources. There are 8 namespace types:\n\n| Namespace | Flag | What It Isolates |\n|-----------|------|------------------|\n| **PID** | `CLONE_NEWPID` | Process IDs - container sees PID 1 |\n| **Network** | `CLONE_NEWNET` | Network interfaces, routing, ports |\n| **Mount** | `CLONE_NEWNS` | Filesystem mount points |\n| **UTS** | `CLONE_NEWUTS` | Hostname and domain name |\n| **IPC** | `CLONE_NEWIPC` | System V IPC, message queues |\n| **User** | `CLONE_NEWUSER` | User and group IDs |\n| **Cgroup** | `CLONE_NEWCGROUP` | Cgroup root directory |\n| **Time** | `CLONE_NEWTIME` | System clocks (Linux 5.6+) |\n\n#### PID Namespace Deep Dive\n\n```bash\n# On the host - see all processes\n$ ps aux | grep nginx\nroot 1234 nginx: master process\nwww-data 1235 nginx: worker process\n\n# Inside the container - different view\n$ docker exec nginx-container ps aux\nPID USER COMMAND\n1 root nginx: master process\n2 www-data nginx: worker process\n```\n\n**The same process has two PIDs:**\n- Host PID: 1234 (real)\n- Container PID: 1 (virtualized)\n\nThis is why `kill 1` inside a container doesn't kill the host's init - different PID namespace.\n\n**PID 1 special responsibilities:**\n1. Reap orphaned child processes (zombie cleanup)\n2. Forward signals to children\n3. Handle SIGTERM gracefully for container shutdown\n\nMost applications aren't designed to be PID 1. Solutions:\n- `docker run --init` - injects tini as PID 1\n- Use dumb-init or tini in your Dockerfile\n- Properly handle signals in your app\n\n#### Network Namespace Deep Dive\n\nEach container gets its own:\n- Network interfaces (eth0, lo)\n- Routing tables\n- iptables rules\n- Port bindings\n\n```bash\n# Host's network interfaces\n$ ip link\n1: lo: \n2: eth0: \n3: docker0: \n4: veth1234@if5: \n\n# Container sees only its interfaces\n$ docker exec container ip link\n1: lo: \n5: eth0@if4: \n```\n\n**How container networking works:**\n\n```\n┌─────────────────────────────────────────────────────┐\n│ HOST │\n│ │\n│ ┌────────────┐ ┌────────────┐ │\n│ │ Container │ │ docker0 │ │\n│ │ │ │ bridge │ │\n│ │ eth0 ─────┼─ veth ──┼─ 172.17.0.1│── eth0 ──────┼──► Internet\n│ │ 172.17.0.2 │ pair │ │ │\n│ │ │ │ │ │\n│ └────────────┘ └────────────┘ │\n│ │\n└─────────────────────────────────────────────────────┘\n```\n\nThe veth pair is like a virtual network cable connecting the container's namespace to the host's docker0 bridge.\n\n#### Mount Namespace Deep Dive\n\nContainers see a different filesystem than the host:\n\n```bash\n# Host root filesystem\n$ ls /\nbin etc home lib root usr var ...\n\n# Container sees its own root\n$ docker exec container ls /\napp bin etc lib tmp ...\n```\n\nThis uses **overlay filesystem** (covered in Part 2), mounted in the container's mount namespace.\n\n**Volumes are bind mounts into the namespace:**\n\n```bash\ndocker run -v /host/data:/container/data nginx\n# Creates bind mount visible only in container's mount namespace\n```\n\n### Control Groups (cgroups): Resource Limits\n\nNamespaces provide isolation. Cgroups provide resource limits.\n\n**What cgroups control:**\n\n| Controller | Limits |\n|------------|--------|\n| `memory` | RAM usage, swap |\n| `cpu` | CPU time allocation |\n| `cpuset` | Which CPUs can be used |\n| `blkio` | Disk I/O bandwidth |\n| `pids` | Maximum number of processes |\n| `devices` | Device access |\n\n#### cgroup v1 vs v2\n\n```bash\n# Check which version you're using\n$ mount | grep cgroup\ncgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)\n\n# v1 uses multiple hierarchies (legacy)\n# v2 uses unified hierarchy (modern, default since 2021)\n```\n\n**v2 advantages:**\n- Single unified hierarchy\n- Cleaner resource distribution\n- Better pressure stall information (PSI)\n- Required for some K8s features\n\n#### Memory Limits in Practice\n\n```bash\n# Set 512MB memory limit\ndocker run -m 512m nginx\n\n# What actually happens:\n$ cat /sys/fs/cgroup/docker//memory.max\n536870912 # 512MB in bytes\n```\n\n**OOM behavior:**\n\n```\n┌─────────────────────────────────────────────────────┐\n│ Container Memory Usage │\n│ │\n│ ┌─────────────────────────────────────────────────┐│\n│ │ ││\n│ │ ████████████████████░░░░░░░░░░░ 450MB ││\n│ │ ││\n│ │ ─────────────────────────────── 512MB limit ││\n│ └─────────────────────────────────────────────────┘│\n│ │\n│ If exceeds limit: │\n│ 1. Kernel tries to reclaim memory │\n│ 2. If can't: OOM killer activates │\n│ 3. Container's main process killed (SIGKILL) │\n│ 4. Exit code: 137 (128 + 9) │\n└─────────────────────────────────────────────────────┘\n```\n\n#### CPU Limits in Practice\n\nTwo main mechanisms:\n\n```bash\n# 1. CPU quota - hard limit\ndocker run --cpus=1.5 nginx\n# Can use at most 1.5 CPUs (150% of one core)\n\n# 2. CPU shares - relative weight\ndocker run --cpu-shares=512 nginx\n# Gets half the weight of default (1024)\n# Only matters under contention\n```\n\n**When to use which:**\n\n| Scenario | Use | Why |\n|----------|-----|-----|\n| Predictable workload | `--cpus` | Guaranteed max |\n| Batch jobs | `--cpu-shares` | Share fairly |\n| Latency-sensitive | `--cpuset-cpus` | Dedicated cores |\n\n### What Happens During `docker run`\n\nLet's trace `docker run -d -p 8080:80 nginx`:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ docker run -d nginx │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ 1. Docker CLI → Docker Daemon (dockerd) │\n│ - Parse arguments │\n│ - Validate image exists (or pull) │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ 2. Docker Daemon → containerd │\n│ - Create container metadata │\n│ - Prepare rootfs (overlay mount) │\n│ - Generate OCI runtime spec (config.json) │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ 3. containerd → containerd-shim → runc │\n│ - shim: Keeps container alive if containerd restarts │\n│ - runc: Actually creates the container │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ 4. runc creates container: │\n│ a. Create namespaces (clone syscall with namespace flags) │\n│ b. Set up cgroups │\n│ c. Configure seccomp filters │\n│ d. Set capabilities │\n│ e. Pivot root to container filesystem │\n│ f. Execute entrypoint (nginx) │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ 5. Container running! │\n│ - Process running in isolated namespaces │\n│ - Resource limits enforced by cgroups │\n│ - containerd-shim monitors for exit │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**Component responsibilities:**\n\n| Component | Role |\n|-----------|------|\n| Docker CLI | User interface, command parsing |\n| dockerd | Image management, networking, volumes |\n| containerd | Container lifecycle, image transfer |\n| containerd-shim | Keep container running across daemon restarts |\n| runc | Actually create container (OCI runtime) |\n\n### The OCI Runtime Specification\n\nOCI (Open Container Initiative) defines standards for containers. The runtime spec defines:\n\n1. **config.json** - Container configuration\n\n```json\n{\n \"ociVersion\": \"1.0.2\",\n \"process\": {\n \"args\": [\"nginx\", \"-g\", \"daemon off;\"],\n \"env\": [\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin\"],\n \"cwd\": \"/\",\n \"user\": { \"uid\": 0, \"gid\": 0 }\n },\n \"root\": {\n \"path\": \"rootfs\",\n \"readonly\": false\n },\n \"linux\": {\n \"namespaces\": [\n { \"type\": \"pid\" },\n { \"type\": \"network\" },\n { \"type\": \"mount\" },\n { \"type\": \"uts\" },\n { \"type\": \"ipc\" }\n ],\n \"resources\": {\n \"memory\": { \"limit\": 536870912 }\n }\n }\n}\n```\n\n2. **rootfs/** - The container's filesystem\n\nThis standardization means you can use different runtimes:\n- **runc** - Default, reference implementation\n- **crun** - Faster, written in C\n- **gVisor (runsc)** - Sandboxed, intercepts syscalls\n- **Kata Containers** - VM-level isolation\n\n### Verifying Isolation from the Host\n\nYou can peek inside namespaces from the host:\n\n```bash\n# Get container's PID\nCONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' nginx)\n\n# See its namespaces\nls -la /proc/$CONTAINER_PID/ns/\n# lrwxrwxrwx 1 root root 0 Dec 11 10:00 net -> net:[4026532456]\n# lrwxrwxrwx 1 root root 0 Dec 11 10:00 pid -> pid:[4026532457]\n# lrwxrwxrwx 1 root root 0 Dec 11 10:00 mnt -> mnt:[4026532454]\n\n# Enter container's network namespace\nnsenter -t $CONTAINER_PID -n ip addr\n\n# View container's cgroup limits\ncat /sys/fs/cgroup/docker/$CONTAINER_ID/memory.max\n\n# See what container sees as root filesystem\nls /proc/$CONTAINER_PID/root/\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Treating Containers Like VMs\n\n```bash\n# ❌ Running SSH inside container\ndocker run -d myapp-with-ssh\n\n# ❌ Running multiple services without supervisor\ndocker run -d \"nginx && postgres && redis\"\n\n# ✅ One process per container\ndocker run -d nginx\ndocker run -d postgres\ndocker run -d redis\n```\n\n**Why it matters:** Containers are process-centric. Multiple processes need proper init, signal handling, and logging becomes complicated.\n\n### Mistake 2: Ignoring PID 1 Responsibilities\n\n```bash\n# ❌ Running app directly as PID 1\nFROM python:3.11\nCMD [\"python\", \"app.py\"]\n# Python doesn't reap zombies, doesn't forward signals\n\n# ✅ Using init\nFROM python:3.11\nRUN apt-get update && apt-get install -y tini\nENTRYPOINT [\"/usr/bin/tini\", \"--\"]\nCMD [\"python\", \"app.py\"]\n\n# ✅ Or use --init flag\ndocker run --init python-app\n```\n\n### Mistake 3: Assuming Container Isolation = Security\n\n```bash\n# ❌ Running as root, assuming container provides security\ndocker run -v /:/host-root dangerous-app\n# Container can access entire host filesystem!\n\n# ❌ Privileged mode\ndocker run --privileged untrusted-app\n# Container has almost full host access\n\n# ✅ Proper security hardening (see Part 12)\ndocker run \\\n --user 1000:1000 \\\n --cap-drop ALL \\\n --read-only \\\n secure-app\n```\n\n---\n\n## 🐛 Debug This: The Invisible Process\n\nA developer reports: \"My container shows PID 1 as my app, but `top` on the host shows it as PID 45678. When I try to `kill 45678` from another container, it says 'No such process'. What's going on?\"\n\n```bash\n# Inside container A\n$ ps aux\nPID COMMAND\n1 /app/server\n\n# On host\n$ ps aux | grep server\nroot 45678 /app/server\n\n# Inside container B\n$ kill 45678\nbash: kill: (45678) - No such process\n```\n\n**Why can't Container B kill the process?**\n\n---\n\n**✅ Solution:**\n\nEach container has its own **PID namespace**. Container B can only see processes in its own namespace - it cannot see or signal processes in Container A's namespace.\n\nThe PID 45678 is the **host PID**. Container B has no visibility into the host's PID namespace (by default). Even if Container B knew the host PID, it couldn't send signals to it.\n\n**Ways to actually kill from Container B:**\n\n```bash\n# Option 1: Share PID namespace (removes isolation!)\ndocker run --pid=container:containerA container-b\n# Now container B sees container A's processes\n\n# Option 2: Use Docker API\ndocker kill container-a\n\n# Option 3: Share host PID namespace (dangerous!)\ndocker run --pid=host container-b\n# Container can now see/kill ALL host processes\n```\n\n**Key insight:** PID namespace isolation is bidirectional. Containers can't see each other's processes, which is both a security feature and a source of confusion.\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Explore Your Container's Namespaces\n⭐ Difficulty: Easy | ⏱️ Time: 15 minutes\n\n```bash\n# 1. Run a container\ndocker run -d --name ns-explore nginx\n\n# 2. Get its PID\ndocker inspect --format '{{.State.Pid}}' ns-explore\n\n# 3. List its namespaces\nsudo ls -la /proc//ns/\n\n# 4. Compare with host namespaces\nsudo ls -la /proc/1/ns/\n\n# Question: Which namespaces are different? Which are shared?\n```\n\n**Expected findings:** PID, mount, UTS, IPC, network namespaces should differ. User namespace may be shared (unless using user namespace remapping).\n\n---\n\n### Exercise 2: Watch OOM Killer in Action\n⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes\n\n```bash\n# 1. Run container with 100MB limit\ndocker run -d --name oom-test -m 100m python:3.11 \\\n python -c \"\nimport time\ndata = []\nwhile True:\n data.append('x' * 10_000_000) # 10MB chunks\n print(f'Allocated {len(data) * 10}MB')\n time.sleep(1)\n\"\n\n# 2. Watch memory usage\ndocker stats oom-test\n\n# 3. Watch for OOM (in another terminal)\ndocker events --filter 'event=oom'\n\n# 4. Check exit code\ndocker inspect --format '{{.State.ExitCode}}' oom-test\n# Should be 137 (128 + SIGKILL)\n\n# 5. Check kernel logs\ndmesg | grep -i \"killed process\"\n```\n\n---\n\n### Exercise 3: Create Container \"By Hand\" with unshare\n⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 30 minutes\n\nCreate a minimal container without Docker using Linux primitives:\n\n```bash\n# 1. Create a minimal rootfs\nmkdir -p /tmp/mini-container/rootfs\ncd /tmp/mini-container\n\n# Copy busybox (static binary, no dependencies)\ncp $(which busybox) rootfs/\ncd rootfs\nfor cmd in sh ls ps mount; do\n ln -s busybox $cmd\ndone\n\n# 2. Create namespaces and chroot\nsudo unshare --mount --uts --ipc --net --pid --fork \\\n chroot /tmp/mini-container/rootfs /sh\n\n# 3. Inside, set up minimal filesystem\nmount -t proc proc /proc # Won't work without /proc dir\nhostname container-test\nps aux # Should show only your shell as PID 1!\n```\n\n**Challenge:** Why doesn't `ps` work? Fix it by creating the necessary directories.\n\n---\n\n### Exercise 4: Measure Namespace Overhead\n⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 25 minutes\n\n```bash\n# Compare process startup times\n\n# 1. Direct process\ntime for i in {1..100}; do /bin/true; done\n\n# 2. With new namespaces (container-like)\ntime for i in {1..100}; do\n unshare --mount --pid --fork /bin/true\ndone\n\n# 3. Full docker run\ntime for i in {1..100}; do\n docker run --rm alpine /bin/true\ndone\n\n# Questions:\n# - How much overhead do namespaces add?\n# - How much does Docker add on top of namespaces?\n```\n\n---\n\n### Exercise 5: Debug Zombie Processes\n⭐⭐⭐⭐ Difficulty: Expert | ⏱️ Time: 30 minutes\n\n```bash\n# 1. Create a container that spawns zombies\ndocker run -d --name zombie-factory python:3.11 \\\n python -c \"\nimport os\nimport time\nwhile True:\n pid = os.fork()\n if pid == 0:\n os._exit(0) # Child exits immediately\n time.sleep(0.5) # Parent doesn't wait()\n\"\n\n# 2. Watch zombie count grow\nwatch -n 1 'docker exec zombie-factory ps aux | grep defunct | wc -l'\n\n# 3. Verify on host\nps aux | grep defunct\n\n# 4. Now fix it - restart with --init\ndocker stop zombie-factory\ndocker run -d --name zombie-fixed --init python:3.11 \\\n python -c \"\nimport os\nimport time\nwhile True:\n pid = os.fork()\n if pid == 0:\n os._exit(0)\n time.sleep(0.5)\n\"\n\n# 5. Verify no zombies accumulate\nwatch -n 1 'docker exec zombie-fixed ps aux | grep defunct | wc -l'\n```\n\n---\n\n## 🎤 Senior-Level Interview Questions\n\n### Q1: What's the difference between a container and a VM?\n\n**Strong Answer:**\n\n\"The fundamental difference is the isolation level. VMs virtualize hardware - each VM runs its own kernel, booting a full OS. The hypervisor manages hardware sharing.\n\nContainers virtualize the OS - they share the host kernel but use Linux namespaces for isolation and cgroups for resource limits. This means:\n\n1. **Startup:** Containers start in milliseconds (just a process), VMs in seconds/minutes (full boot)\n2. **Overhead:** Containers use MBs of memory overhead, VMs use GBs\n3. **Isolation:** VMs have stronger isolation (separate kernel). A kernel vulnerability affects all containers but only one VM\n4. **Density:** You can run 100s of containers on a host, maybe 10s of VMs\n\nFor most workloads, containers are better for efficiency. VMs are better when you need strong multi-tenant isolation or different operating systems.\"\n\n### Q2: A container process shows as PID 1 inside but PID 5000 on the host. Explain.\n\n**Strong Answer:**\n\n\"This is PID namespace isolation. When Docker creates a container, it uses the `CLONE_NEWPID` flag with the clone syscall to create a new PID namespace.\n\nInside this namespace, PIDs start from 1 - the first process becomes PID 1. But the kernel still tracks the 'real' PID in the root namespace.\n\nThe process has two PIDs simultaneously:\n- Namespace PID: 1 (what the container sees)\n- Host PID: 5000 (what the kernel uses)\n\nThis matters because PID 1 has special responsibilities: signal handling and zombie reaping. Most applications aren't designed for this, which is why we often use `--init` to run tini as PID 1 instead.\"\n\n### Q3: How do you debug a container that exits immediately with code 137?\n\n**Strong Answer:**\n\n\"Exit code 137 is 128 + 9, meaning the process received SIGKILL. This is almost always the OOM killer.\n\nMy debugging steps:\n\n1. **Confirm OOM:** `docker inspect` shows `OOMKilled: true`\n2. **Check memory limit:** Was it set too low? `docker inspect --format '{{.HostConfig.Memory}}'`\n3. **Check actual usage:** `docker stats` before it dies, or kernel logs: `dmesg | grep -i 'killed process'`\n4. **Analyze the app:** Is it a memory leak, or does it legitimately need more?\n5. **Fix options:**\n - Increase limit if app needs it\n - Fix memory leak if there is one\n - For JVM: ensure `-XX:+UseContainerSupport` and set `-XX:MaxRAMPercentage`\n - Add memory monitoring/alerting before OOM\n\nThe key insight: OOM kills are silent from the container's perspective. No logs, no graceful shutdown, just SIGKILL.\"\n\n### Q4: Why might a process inside a container not see other processes also inside that container?\n\n**Strong Answer:**\n\n\"If a container is running multiple processes and one can't see the others, there are a few possibilities:\n\n1. **Different PID namespaces:** If a process was started with additional isolation, it might have its own PID namespace. Docker `--init` does this correctly, but custom setups might not.\n\n2. **Proc filesystem not mounted:** If `/proc` isn't mounted (or mounted incorrectly), `ps` won't show processes. This happens in minimal containers.\n\n3. **Incorrect /proc mount:** If `/proc` was mounted from outside the namespace, it shows the wrong namespace's processes.\n\n4. **Process actually died:** Race condition - process started but exited before `ps` ran.\n\nDebugging: Check `/proc` is mounted correctly with `mount | grep proc`. Use `ls /proc` to see if PIDs are visible. Check if there are multiple PID namespaces with `lsns -t pid`.\"\n\n### Q5: Explain what happens between `docker run nginx` and nginx actually serving requests.\n\n**Strong Answer:**\n\n\"The full sequence involves multiple components:\n\n1. **Docker CLI** parses the command, communicates with Docker daemon via REST API over Unix socket\n\n2. **Docker daemon (dockerd):**\n - Checks image exists locally, pulls if not\n - Creates container metadata\n - Sets up networking (creates veth pair, connects to bridge)\n - Prepares storage (overlay mount)\n\n3. **Containerd:**\n - Receives request from dockerd\n - Creates OCI runtime bundle (config.json + rootfs)\n - Spawns containerd-shim\n\n4. **Containerd-shim:**\n - Stays running to monitor container\n - Survives containerd restarts\n - Calls runc\n\n5. **Runc (OCI runtime):**\n - Creates namespaces (clone syscall)\n - Sets up cgroups\n - Applies seccomp filters\n - Sets capabilities\n - Performs pivot_root to container's filesystem\n - Execs nginx\n\n6. **Nginx starts** as PID 1 in container (PID N on host)\n\n7. **Network path:**\n - Nginx binds to port 80 inside container\n - Docker sets up iptables DNAT rule\n - External traffic to host:8080 → container:80\n\nTotal time: ~100-500ms for a cached image.\"\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Core Concepts\n\n| Concept | Key Point |\n|---------|-----------|\n| Containers | Processes with isolation, not VMs |\n| Namespaces | Provide resource isolation (PID, network, mount, etc.) |\n| Cgroups | Provide resource limits (memory, CPU, I/O) |\n| PID 1 | Has special responsibilities - use --init |\n| OCI Runtime | Standardized container creation (runc) |\n\n### The Container Equation\n\n```\nContainer = Linux Process\n + Namespaces (isolation)\n + Cgroups (resource limits)\n + Overlay filesystem (image)\n```\n\n### What You Can Do Now\n\n1. **Debug namespace issues:** Use `nsenter` and `lsns` to inspect containers\n2. **Understand resource limits:** Know why OOM happens (exit code 137)\n3. **Avoid zombie processes:** Use `--init` or proper signal handling\n4. **Explain containers technically:** Move beyond \"lightweight VMs\" explanation\n\n---\n\n## 📋 Quick Reference\n\n### Namespace Types\n\n| Namespace | Creates Isolation For |\n|-----------|----------------------|\n| pid | Process IDs |\n| net | Network stack |\n| mnt | Mount points |\n| uts | Hostname |\n| ipc | IPC resources |\n| user | User/group IDs |\n| cgroup | Cgroup root |\n| time | System clocks |\n\n### Key Commands\n\n```bash\n# View container namespaces\nlsns -p $(docker inspect --format '{{.State.Pid}}' CONTAINER)\n\n# Enter container namespace\nnsenter -t PID -n -m -u -i -p /bin/sh\n\n# View cgroup limits\ncat /sys/fs/cgroup/docker/CONTAINER_ID/memory.max\n\n# Check for OOM\ndocker inspect --format '{{.State.OOMKilled}}' CONTAINER\n\n# View OCI config\ndocker inspect --format '{{json .HostConfig}}' CONTAINER | jq\n```\n\n### Exit Codes\n\n| Code | Signal | Meaning |\n|------|--------|---------|\n| 0 | - | Normal exit |\n| 1 | - | Application error |\n| 137 | SIGKILL (9) | OOM killed or `docker kill` |\n| 143 | SIGTERM (15) | Graceful shutdown |\n\n---\n\n## 📅 Review Schedule\n\n| Day | Task | Time |\n|-----|------|------|\n| Day 1 | Re-read namespace section, draw diagram from memory | 10 min |\n| Day 3 | Do Exercise 1 (explore namespaces) | 15 min |\n| Day 7 | Explain container vs VM to a colleague | 5 min |\n| Day 14 | Do Exercise 2 (OOM killer) | 20 min |\n| Day 30 | Answer interview questions without looking | 15 min |\n\n---\n\n## 📚 Series Navigation\n\n| Previous | Current | Next |\n|----------|---------|------|\n| [Part 0: How to Use This Series](docker-part-0-how-to-use-this-series) | **Part 1: Container Internals** | [Part 2: Image Anatomy](docker-part-2-image-anatomy) |\n\n**Docker Compendium Series:**\n- [Part 0: How to Use This Series](docker-part-0-how-to-use-this-series)\n- **Part 1: Container Internals** ← You are here\n- [Part 2: Image Anatomy](docker-part-2-image-anatomy)\n- [Part 3: Build Process Deep Dive](docker-part-3-build-process)\n- [Part 4: Networking Internals](docker-part-4-networking-internals)\n","featuredImage","isPublished","isFeatured","subscriptionType","BASIC","publishedAt","2026-06-10T07:19:25.374663Z","createdAt","2026-06-10T07:19:25.375338Z","updatedAt","2026-06-16T21:26:56.086200Z","readingTimeMinutes",18,"estimatedReadingTime","authorId",3,"authorUsername","tomasz_main","categoryName","Devops","readingTime","seriesId","seriesOrder","author",{"_33":57,"_99":59,"_100":101,"_102":103,"_104":-5,"_105":-5,"_106":-5,"_107":-5,"_108":-5},"category",{"_33":95,"_73":61,"_37":96,"_76":97,"_98":-5},"tags",[79,80,81,82],"series",{"_33":57,"_73":74,"_37":75,"_76":77,"_78":24},"name","Docker Compendium: From Practitioner to Expert","docker-compendium","description","Master Docker internals, optimization patterns, and production best practices. For developers who use Docker daily but want deeper understanding of what's happening under the hood. Not a beginner series - we assume you can write Dockerfiles and use docker-compose. Learn what namespaces and cgroups really do, why your builds are slow, how to debug containers that crash, and how to run Docker in production like a pro.\n","isActive",{"_33":92,"_73":93,"_37":94},{"_33":89,"_73":90,"_37":91},{"_33":86,"_73":87,"_37":88},{"_33":83,"_73":84,"_37":85},219,"Namespaces","namespaces",217,"Linux","linux",213,"Docker","docker",220,"Cgroups","cgroups",7,"devops","Category for Devops","colorHex","username","firstName","Tomasz","lastName","Jarosz","profilePicture","bio","website","githubUrl","linkedinUrl",59,"Container Resource Management","docker-part-9-resource-management","Memory limits, CPU allocation, OOM killer, JVM tuning","# Docker Compendium Part 9: Container Resource Management\n\nYour Java app needs 512MB but uses 2GB. Your Node.js service gets CPU throttled mysteriously. Containers die with no logs. This article explains how Docker limits resources, how to size containers correctly, and how to avoid the OOM killer.\n\n## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| Topic | Memory limits, CPU allocation, OOM killer, JVM/Node.js tuning |\n| Complexity | Advanced |\n| Prerequisites | Part 1 (Container Internals - cgroups) |\n| Key Insight | Container limits are enforced by kernel cgroups, not the application |\n| Time to Master | 3-4 hours |\n\n---\n\n## 🎯 What You'll Learn\n\n- **Memory management** - hard limits, soft limits, swap, OOM behavior\n- **CPU management** - shares, quota, pinning, throttling\n- **Language-specific tuning** - JVM, Node.js, Python in containers\n- **Right-sizing** - how to determine appropriate limits\n- **Monitoring** - detecting resource problems before they cause outages\n\n---\n\n## 🔥 Production Story: The OOM Serial Killer\n\nA team deployed microservices to Kubernetes. Pods kept restarting randomly - some after hours, some after minutes. No logs, no errors, just disappeared.\n\n**Investigation:**\n\n```bash\n$ kubectl describe pod api-server\n...\nLast State: Terminated\n Reason: OOMKilled\n Exit Code: 137\n\n$ kubectl get events\nWarning OOMKilling 4m kernel-monitor Memory cgroup out of memory: Kill process 1234\n```\n\n**The setup:**\n\n```yaml\nresources:\n limits:\n memory: \"512Mi\"\n requests:\n memory: \"256Mi\"\n```\n\n```dockerfile\nFROM eclipse-temurin:21\nCOPY app.jar /app.jar\nCMD [\"java\", \"-jar\", \"/app.jar\"]\n```\n\n**Root cause:** JVM default heap is 25% of **physical memory** (host's 32GB), not container limit. JVM tried to allocate 8GB heap inside 512MB container.\n\n**The fix:**\n\n```dockerfile\nFROM eclipse-temurin:21\nCOPY app.jar /app.jar\n# Use container-aware memory settings\nCMD [\"java\", \\\n \"-XX:+UseContainerSupport\", \\\n \"-XX:MaxRAMPercentage=75.0\", \\\n \"-jar\", \"/app.jar\"]\n```\n\nNow JVM allocates 75% of 512MB = 384MB heap, leaving room for metaspace, threads, and OS overhead.\n\n**Lesson:** Applications don't automatically respect container limits. You must configure them to be container-aware.\n\n---\n\n## 🧠 Mental Model: Resource Limits Stack\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│ CONTAINER RESOURCE LIMITS │\n│ │\n│ ┌──────────────────────────────────────────────────────────────────────┐\n│ │ APPLICATION LAYER │\n│ │ │\n│ │ JVM: -Xmx, -XX:MaxRAMPercentage │\n│ │ Node: --max-old-space-size │\n│ │ Python: Various pool sizes │\n│ │ │\n│ │ ⚠️ App may try to use MORE than container allows! │\n│ └──────────────────────────────────────────────────────────────────────┘\n│ │ │\n│ ▼ │\n│ ┌──────────────────────────────────────────────────────────────────────┐\n│ │ CONTAINER LAYER │\n│ │ │\n│ │ docker run --memory=512m --cpus=1.5 │\n│ │ │\n│ │ These translate to cgroup limits in kernel │\n│ └──────────────────────────────────────────────────────────────────────┘\n│ │ │\n│ ▼ │\n│ ┌──────────────────────────────────────────────────────────────────────┐\n│ │ CGROUP LAYER │\n│ │ │\n│ │ memory.max = 536870912 (512MB) │\n│ │ cpu.max = 150000 100000 (1.5 CPUs) │\n│ │ │\n│ │ Kernel ENFORCES these limits │\n│ └──────────────────────────────────────────────────────────────────────┘\n│ │ │\n│ ▼ │\n│ ┌──────────────────────────────────────────────────────────────────────┐\n│ │ KERNEL ACTIONS │\n│ │ │\n│ │ Memory exceeded → OOM Killer → SIGKILL (exit 137) │\n│ │ CPU exceeded → Throttling (slower, not killed) │\n│ │ │\n│ └──────────────────────────────────────────────────────────────────────┘\n│ │\n└─────────────────────────────────────────────────────────────────────────┘\n```\n\n---\n\n## 🔬 Deep Dive\n\n### Memory Limits\n\n**Setting memory limits:**\n\n```bash\n# Hard limit - container killed if exceeded\ndocker run --memory=512m nginx\n\n# Soft limit (reservation) - for orchestrator scheduling\ndocker run --memory=512m --memory-reservation=256m nginx\n\n# Memory + swap limit\ndocker run --memory=512m --memory-swap=1g nginx\n# Container can use 512MB RAM + 512MB swap\n\n# Memory only (disable swap)\ndocker run --memory=512m --memory-swap=512m nginx\n# Or equivalently:\ndocker run --memory=512m --memory-swappiness=0 nginx\n```\n\n**What counts toward memory limit:**\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ CONTAINER MEMORY │\n│ │\n│ ┌────────────────────────────────────────────────────────────┐ │\n│ │ Application Heap │ │\n│ │ (Java: -Xmx, Node: --max-old-space-size) │ │\n│ ├────────────────────────────────────────────────────────────┤ │\n│ │ Application Off-Heap │ │\n│ │ (JVM metaspace, native memory, thread stacks) │ │\n│ ├────────────────────────────────────────────────────────────┤ │\n│ │ Kernel Buffers & Cache │ │\n│ │ (File system cache, network buffers) │ │\n│ ├────────────────────────────────────────────────────────────┤ │\n│ │ Shared Libraries │ │\n│ │ (libc, language runtime, etc.) │ │\n│ └────────────────────────────────────────────────────────────┘ │\n│ │\n│ Total counted against cgroup memory limit │\n│ │\n│ ⚠️ Kernel cache is usually reclaimable, │\n│ but still counts until reclaimed! │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**OOM Killer behavior:**\n\n```bash\n# Container exceeds memory limit\n# 1. Kernel tries to reclaim memory (cache, etc.)\n# 2. If still over limit: OOM killer triggers\n# 3. SIGKILL sent to container's main process\n# 4. Exit code: 137 (128 + SIGKILL signal 9)\n\n# Detect OOM\n$ docker inspect container --format '{{.State.OOMKilled}}'\ntrue\n\n# Check kernel logs\n$ dmesg | grep -i \"killed process\"\nMemory cgroup out of memory: Killed process 1234 (java) total-vm:...\n```\n\n### CPU Limits\n\nDocker provides multiple ways to limit CPU:\n\n**1. CPU quota (--cpus) - Hard limit:**\n\n```bash\n# Limit to 1.5 CPUs (150% of one core)\ndocker run --cpus=1.5 myapp\n\n# This translates to:\n# cpu.max = 150000 100000\n# (150000 microseconds per 100000 microsecond period)\n```\n\n**2. CPU shares (--cpu-shares) - Relative weight:**\n\n```bash\n# Default share is 1024\n# This container gets half the CPU of default\ndocker run --cpu-shares=512 myapp\n\n# Only matters under contention!\n# If no contention, container can use all available CPU\n```\n\n**3. CPU pinning (--cpuset-cpus) - Dedicated cores:**\n\n```bash\n# Only use CPU cores 0 and 1\ndocker run --cpuset-cpus=0,1 myapp\n\n# Useful for:\n# - NUMA-aware applications\n# - Latency-sensitive workloads\n# - Preventing core hopping\n```\n\n**When to use which:**\n\n| Scenario | Use | Why |\n|----------|-----|-----|\n| Predictable workload | `--cpus` | Guaranteed limit |\n| Batch jobs | `--cpu-shares` | Fair sharing |\n| Latency-sensitive | `--cpuset-cpus` | Cache locality |\n| Resource guarantee | `--cpus` + `--cpu-shares` | Both limit and priority |\n\n**CPU throttling:**\n\n```bash\n# Unlike memory, CPU doesn't kill - it throttles\n# Container is paused until next period\n\n# Detect throttling\n$ cat /sys/fs/cgroup/docker//cpu.stat\nusage_usec 1234567890\nuser_usec 1000000000\nsystem_usec 234567890\nnr_periods 10000\nnr_throttled 500 # <-- Throttling count\nthrottled_usec 50000000 # <-- Total throttle time\n```\n\n### JVM Container Tuning\n\nModern JVMs (8u191+, 11+) are container-aware but need configuration:\n\n```dockerfile\nFROM eclipse-temurin:21\n\n# Enable container support (default in modern JVMs)\nENV JAVA_OPTS=\"-XX:+UseContainerSupport\"\n\n# Set heap as percentage of container limit\nENV JAVA_OPTS=\"$JAVA_OPTS -XX:MaxRAMPercentage=75.0\"\nENV JAVA_OPTS=\"$JAVA_OPTS -XX:InitialRAMPercentage=50.0\"\n\n# Or set exact limits\n# ENV JAVA_OPTS=\"$JAVA_OPTS -Xms256m -Xmx384m\"\n\nCMD [\"sh\", \"-c\", \"java $JAVA_OPTS -jar /app.jar\"]\n```\n\n**Memory breakdown for JVM in container:**\n\n```\nContainer: 512MB\n├── Heap: 75% = 384MB (-XX:MaxRAMPercentage=75.0)\n├── Metaspace: ~64MB (class metadata)\n├── Thread stacks: ~1MB per thread\n├── Native memory: varies\n├── JIT code cache: ~48MB\n└── Other: GC, buffers\n\n⚠️ If heap = container limit, you WILL OOM\n Always leave 20-30% for non-heap\n```\n\n**Checking JVM settings:**\n\n```bash\ndocker run --rm -m 512m eclipse-temurin:21 java \\\n -XX:+UseContainerSupport \\\n -XX:MaxRAMPercentage=75.0 \\\n -XshowSettings:vm -version\n\n# Output shows:\n# Max. Heap Size (Estimated): 384.00M\n```\n\n### Node.js Container Tuning\n\nNode.js has its own memory management:\n\n```dockerfile\nFROM node:20-slim\n\n# Set heap size (default is ~1.4GB on 64-bit)\nENV NODE_OPTIONS=\"--max-old-space-size=384\"\n\n# For container-based auto-sizing (Node 19+)\n# ENV NODE_OPTIONS=\"--max-old-space-size=$(($MEMORY_LIMIT * 75 / 100))\"\n\nCMD [\"node\", \"index.js\"]\n```\n\n**Calculating Node.js memory:**\n\n```bash\n# Container limit: 512MB\n# Node heap: 75% = 384MB\n# System/buffers: 25% = 128MB\n\ndocker run -m 512m -e NODE_OPTIONS=\"--max-old-space-size=384\" mynode\n```\n\n**UV_THREADPOOL_SIZE for I/O-bound apps:**\n\n```dockerfile\n# Default is 4 threads for fs/dns operations\n# Increase for high I/O workloads\nENV UV_THREADPOOL_SIZE=16\n```\n\n### Python Container Tuning\n\nPython's memory management is more automatic, but:\n\n```dockerfile\nFROM python:3.11-slim\n\n# Disable bytecode writing\nENV PYTHONDONTWRITEBYTECODE=1\n\n# Unbuffered output (for logging)\nENV PYTHONUNBUFFERED=1\n\n# Memory allocator (can reduce memory usage)\nENV PYTHONMALLOC=malloc\n\nCMD [\"python\", \"app.py\"]\n```\n\n**Memory-intensive Python (NumPy, Pandas):**\n\n```bash\n# Limit OpenBLAS threads\nENV OPENBLAS_NUM_THREADS=1\nENV OMP_NUM_THREADS=1\n\n# For scikit-learn\nENV JOBLIB_MULTIPROCESSING=0\n```\n\n### Monitoring Container Resources\n\n**Real-time stats:**\n\n```bash\n# Docker stats\ndocker stats\n# CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O\n# app 45.2% 384MiB / 512MiB 75% 1.5MB / 800KB\n\n# Specific container\ndocker stats app --no-stream --format \"{{.MemUsage}} {{.MemPerc}}\"\n```\n\n**Inspect cgroup limits:**\n\n```bash\n# Memory limit\ncat /sys/fs/cgroup/docker//memory.max\n\n# Current memory usage\ncat /sys/fs/cgroup/docker//memory.current\n\n# CPU limit\ncat /sys/fs/cgroup/docker//cpu.max\n\n# CPU throttling stats\ncat /sys/fs/cgroup/docker//cpu.stat\n```\n\n**Prometheus metrics:**\n\n```yaml\n# docker-compose with metrics\nservices:\n app:\n image: myapp\n deploy:\n resources:\n limits:\n memory: 512M\n cpus: '1.5'\n\n cadvisor:\n image: gcr.io/cadvisor/cadvisor\n volumes:\n - /:/rootfs:ro\n - /var/run:/var/run:ro\n - /sys:/sys:ro\n - /var/lib/docker/:/var/lib/docker:ro\n ports:\n - \"8080:8080\"\n```\n\n### Right-Sizing Containers\n\n**Process for determining limits:**\n\n```\n1. Profile application under load\n └─ Measure peak memory, average CPU\n\n2. Add safety margin\n └─ Memory: +20-30% for GC, buffers\n └─ CPU: Consider burst capacity\n\n3. Test under various conditions\n └─ Cold start\n └─ Peak load\n └─ Memory leak scenarios\n\n4. Monitor in production\n └─ Actual usage vs limits\n └─ Throttling frequency\n └─ OOM events\n\n5. Iterate\n └─ Adjust based on real data\n```\n\n**Common sizing guidelines:**\n\n| App Type | Memory Suggestion | CPU Suggestion |\n|----------|-------------------|----------------|\n| JVM microservice | 512MB - 1GB | 0.5 - 2 |\n| Node.js API | 256MB - 512MB | 0.25 - 1 |\n| Python web | 256MB - 512MB | 0.25 - 1 |\n| ML inference | 2GB - 8GB | 1 - 4 |\n| Static content | 64MB - 128MB | 0.1 - 0.25 |\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Setting Heap = Container Limit\n\n```bash\n# ❌ JVM tries 512MB heap in 512MB container\ndocker run -m 512m myapp -Xmx512m\n# Result: OOM - no room for non-heap memory!\n\n# ✅ Leave room for non-heap\ndocker run -m 512m myapp -Xmx384m\n# Or use percentage\ndocker run -m 512m myapp -XX:MaxRAMPercentage=75.0\n```\n\n### Mistake 2: Ignoring CPU Throttling\n\n```bash\n# App seems slow but no errors\n# Check for throttling:\n$ cat /sys/fs/cgroup/docker//cpu.stat\nnr_throttled 15000 # <-- Throttled 15000 times!\n\n# Solution: Increase CPU limit or optimize app\ndocker run --cpus=2.0 myapp # Was --cpus=0.5\n```\n\n### Mistake 3: Not Setting Any Limits\n\n```bash\n# ❌ No limits - can consume entire host\ndocker run myapp\n\n# ✅ Always set limits in production\ndocker run --memory=512m --cpus=1.0 myapp\n```\n\n---\n\n## 🐛 Debug This: The Mysterious Slow Container\n\nA developer reports: \"My container runs fine with 2 CPUs but is incredibly slow with 0.5 CPU. It's not even using all 0.5 CPU according to docker stats!\"\n\n```bash\n$ docker run --cpus=0.5 myapp\n# App very slow\n\n$ docker stats myapp\nCONTAINER CPU % MEM USAGE\nmyapp 35% 256MiB\n\n# CPU only 35% - not even hitting limit?\n```\n\n**Why is the app slow if it's not hitting CPU limit?**\n\n---\n\n**✅ Solution:**\n\nThe app is being **throttled**, not fully utilizing the CPU.\n\n**Understanding CPU throttling:**\n\n`--cpus=0.5` means the container can use 50ms of CPU time per 100ms period. If the app does burst computation:\n\n```\nPeriod 1 (0-100ms):\n├── 0-50ms: App runs at 100% (using its quota)\n├── 50-100ms: App THROTTLED (waiting for next period)\n└── Apparent usage: 50% of period, but app waited 50ms!\n\nPeriod 2 (100-200ms):\n├── 100-150ms: App runs again\n├── 150-200ms: Throttled again\n└── And so on...\n```\n\n**docker stats shows average** - 35% average doesn't show the throttling pattern.\n\n**Diagnosis:**\n\n```bash\n# Check throttling stats\ncat /sys/fs/cgroup/docker//cpu.stat\nnr_throttled 10000 # App was throttled 10000 times!\nthrottled_usec 500000000 # 500 seconds total throttle time\n```\n\n**Solutions:**\n\n1. **Increase CPU limit** if app needs it:\n ```bash\n docker run --cpus=1.0 myapp\n ```\n\n2. **Use CPU shares instead** if you want burst capability:\n ```bash\n docker run --cpu-shares=512 myapp\n # Can burst to more CPU if available\n ```\n\n3. **Optimize the app** - reduce CPU-intensive operations\n\n4. **Adjust period** (advanced):\n ```bash\n docker run --cpu-period=50000 --cpu-quota=25000 myapp\n # Same 0.5 CPU but smaller periods = less latency\n ```\n\n**Key insight:** CPU throttling causes latency, not just reduced throughput. A burst-y application can feel much slower than CPU % suggests.\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Observe OOM Killer\n⭐ Difficulty: Easy | ⏱️ Time: 15 minutes\n\n```bash\n# Create memory-hungry app\ndocker run -d --name oom-test -m 100m python:3.11 python -c \"\nimport time\ndata = []\nwhile True:\n data.append('x' * 10_000_000)\n print(f'Allocated {len(data) * 10}MB')\n time.sleep(0.5)\n\"\n\n# Watch it get killed\ndocker logs -f oom-test\n\n# Check OOM status\ndocker inspect oom-test --format '{{.State.OOMKilled}}'\n\n# Check exit code (should be 137)\ndocker inspect oom-test --format '{{.State.ExitCode}}'\n\n# Check kernel logs\ndmesg | tail -20 | grep -i oom\n```\n\n---\n\n### Exercise 2: Measure CPU Throttling\n⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes\n\n```bash\n# CPU-intensive task\ndocker run -d --name cpu-test --cpus=0.5 alpine sh -c \"\nwhile true; do\n # CPU-intensive operation\n dd if=/dev/urandom bs=1M count=10 | md5sum > /dev/null\n sleep 0.1\ndone\n\"\n\n# Monitor CPU usage\ndocker stats cpu-test\n\n# Check throttling\nCONTAINER_ID=$(docker inspect cpu-test --format '{{.Id}}')\ncat /sys/fs/cgroup/docker/${CONTAINER_ID}/cpu.stat\n\n# Compare with higher limit\ndocker stop cpu-test\ndocker run -d --name cpu-test2 --cpus=2.0 alpine sh -c \"\nwhile true; do\n dd if=/dev/urandom bs=1M count=10 | md5sum > /dev/null\n sleep 0.1\ndone\n\"\n\n# Check throttling for this one\nCONTAINER_ID2=$(docker inspect cpu-test2 --format '{{.Id}}')\ncat /sys/fs/cgroup/docker/${CONTAINER_ID2}/cpu.stat\n\n# Compare nr_throttled between them\n```\n\n---\n\n### Exercise 3: JVM Container Sizing\n⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes\n\n```bash\n# Test JVM memory detection\ndocker run --rm -m 512m eclipse-temurin:21 java \\\n -XX:+PrintFlagsFinal -version 2>&1 | grep -E \"MaxHeapSize|InitialHeapSize\"\n\n# With container support (should see ~128MB = 25% of 512MB)\ndocker run --rm -m 512m eclipse-temurin:21 java \\\n -XX:+UseContainerSupport \\\n -XX:+PrintFlagsFinal -version 2>&1 | grep MaxHeapSize\n\n# With 75% percentage\ndocker run --rm -m 512m eclipse-temurin:21 java \\\n -XX:+UseContainerSupport \\\n -XX:MaxRAMPercentage=75.0 \\\n -XshowSettings:vm -version\n\n# Verify with actual app\ncat > /tmp/MemTest.java << 'EOF'\npublic class MemTest {\n public static void main(String[] args) {\n Runtime rt = Runtime.getRuntime();\n System.out.println(\"Max Memory: \" + rt.maxMemory() / 1024 / 1024 + \"MB\");\n System.out.println(\"Total Memory: \" + rt.totalMemory() / 1024 / 1024 + \"MB\");\n System.out.println(\"Free Memory: \" + rt.freeMemory() / 1024 / 1024 + \"MB\");\n }\n}\nEOF\n\ndocker run --rm -m 512m -v /tmp:/code eclipse-temurin:21 sh -c \"\ncd /code && javac MemTest.java && java -XX:MaxRAMPercentage=75.0 MemTest\n\"\n```\n\n---\n\n### Exercise 4: Find the Right Limit\n⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 30 minutes\n\n```bash\n# Profile an app to find appropriate limits\n\n# Create test app with variable load\ncat > /tmp/app.py << 'EOF'\nimport time\nimport sys\nimport random\n\ndata = []\ncpu_work = 0\n\nwhile True:\n # Simulate memory growth\n if random.random() > 0.7:\n data.append('x' * 1_000_000)\n if len(data) > 100:\n data.pop(0)\n\n # Simulate CPU work\n if random.random() > 0.5:\n cpu_work = sum(range(100000))\n\n time.sleep(0.1)\n print(f\"Memory chunks: {len(data)}, CPU result: {cpu_work}\", flush=True)\nEOF\n\n# Run without limits first\ndocker run -d --name profile-test -v /tmp/app.py:/app.py python:3.11-slim python /app.py\n\n# Monitor for 1 minute\ndocker stats profile-test --no-stream\nsleep 60\ndocker stats profile-test --no-stream\n\n# Record peak memory and average CPU\n# Then set limits with 20% buffer\n\n# Clean up\ndocker stop profile-test && docker rm profile-test\n\n# Test with limits\n# docker run -d --name limited-test -m --cpus= ...\n```\n\n---\n\n### Exercise 5: Complete Resource Configuration\n⭐⭐⭐⭐ Difficulty: Expert | ⏱️ Time: 30 minutes\n\nCreate a production-ready docker-compose with proper resource management:\n\n```yaml\n# Requirements:\n# 1. Java API service (512MB, 1 CPU)\n# 2. Node.js worker (256MB, 0.5 CPU)\n# 3. Python data processor (1GB, 2 CPU)\n# 4. Redis cache (128MB, 0.25 CPU)\n# 5. PostgreSQL (512MB, 1 CPU)\n\n# Each service should have:\n# - Memory limit\n# - Memory reservation (50% of limit)\n# - CPU limit\n# - Proper JVM/Node flags where applicable\n# - Health checks\n\n# Test your configuration:\n# - All services start successfully\n# - Services stay within limits under load\n# - No OOM kills during normal operation\n```\n\n---\n\n## 🎤 Senior-Level Interview Questions\n\n### Q1: Explain the difference between --memory and --memory-reservation.\n\n**Strong Answer:**\n\n\"These are hard vs soft limits:\n\n**--memory (hard limit):**\n- Enforced by kernel cgroups\n- If exceeded: OOM killer terminates the container\n- Container cannot use more than this\n\n**--memory-reservation (soft limit):**\n- Scheduling hint for orchestrators\n- Not enforced when memory is available\n- When host is under pressure, kernel tries to reclaim down to reservation\n- Container can exceed this if host has free memory\n\n**Example:**\n```bash\ndocker run --memory=512m --memory-reservation=256m myapp\n```\n\nThe container:\n- Guaranteed 256MB (orchestrator won't over-commit below this)\n- Can burst to 512MB when available\n- Is killed if it tries to exceed 512MB\n\n**Use case:** In Kubernetes, `requests` maps to reservation, `limits` maps to hard limit. Set requests to typical usage, limits to peak + buffer.\n\nI typically set reservation to 50-70% of limit for services with variable memory patterns.\"\n\n### Q2: A Java container keeps getting OOM killed despite setting -Xmx512m in a 512MB container. Why?\n\n**Strong Answer:**\n\n\"JVM uses more than just heap memory. The OOM is because total JVM memory exceeds container limit.\n\n**JVM memory components:**\n\n```\nTotal JVM Memory = Heap (-Xmx)\n + Metaspace (class metadata, ~64MB+)\n + Thread stacks (~1MB per thread)\n + Code cache (JIT compiled code, ~48MB)\n + Native memory (NIO buffers, JNI)\n + GC overhead\n```\n\n**With -Xmx512m:**\n- Heap: 512MB\n- Metaspace: 64MB\n- 50 threads: 50MB\n- Code cache: 48MB\n- Other: varies\n- **Total: ~700MB** in a 512MB container = OOM\n\n**Solutions:**\n\n1. **Reduce heap, leave room for overhead:**\n ```bash\n -Xmx384m # Leave 128MB for non-heap\n ```\n\n2. **Use container-aware settings:**\n ```bash\n -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0\n ```\n JVM calculates 75% of container limit automatically.\n\n3. **Limit other areas:**\n ```bash\n -XX:MaxMetaspaceSize=64m\n -XX:ReservedCodeCacheSize=32m\n -Xss256k # Thread stack size\n ```\n\nRule of thumb: Heap should be 65-75% of container limit maximum.\"\n\n### Q3: How do you detect and handle CPU throttling?\n\n**Strong Answer:**\n\n\"CPU throttling happens when a container exceeds its CPU quota within a scheduling period.\n\n**Detection:**\n\n```bash\n# Check throttling stats\ncat /sys/fs/cgroup/docker//cpu.stat\nnr_throttled 10000 # Times throttled\nthrottled_usec 500000000 # Total throttle time in microseconds\n```\n\n**Symptoms:**\n- High latency spikes\n- Lower than expected throughput\n- docker stats showing less CPU usage than limit\n\n**Monitoring:**\n- Prometheus: `container_cpu_cfs_throttled_periods_total`\n- Alert when throttling rate exceeds threshold\n\n**Solutions:**\n\n1. **Increase CPU limit** if legitimate need:\n ```bash\n --cpus=2.0 # Was 0.5\n ```\n\n2. **Adjust period for lower latency:**\n ```bash\n --cpu-period=50000 --cpu-quota=25000\n # Same 0.5 CPU but smaller periods = less waiting\n ```\n\n3. **Use CPU shares for burst capability:**\n ```bash\n --cpu-shares=512\n # Can burst when other containers are idle\n ```\n\n4. **Optimize application:**\n - Reduce CPU-intensive operations\n - Add caching\n - Async processing\n\nFor latency-sensitive services, I prefer slightly over-provisioning CPU to avoid any throttling. For batch jobs, throttling is acceptable.\"\n\n### Q4: How do you right-size containers for a new application?\n\n**Strong Answer:**\n\n\"I follow a data-driven process:\n\n**Phase 1: Profile without limits**\n```bash\ndocker run myapp # No limits\ndocker stats --no-stream # Peak values over time\n```\n\n**Phase 2: Analyze memory**\n- Peak memory usage\n- Memory growth pattern\n- GC behavior (for JVM)\n\n**Phase 3: Analyze CPU**\n- Average utilization\n- Peak spikes\n- Throttling sensitivity\n\n**Phase 4: Calculate limits**\n```\nMemory limit = Peak usage * 1.2 to 1.3 (20-30% buffer)\nMemory reservation = Average usage\n\nCPU limit = Peak requirement + 20% buffer\nCPU request = Average usage\n```\n\n**Phase 5: Test under various conditions**\n- Cold start\n- Sustained load\n- Spike load\n- Memory leak scenarios\n\n**Phase 6: Monitor and iterate**\n- Track actual usage vs limits\n- Adjust based on real production data\n- Set up alerts for approaching limits\n\n**Example output:**\n```yaml\nresources:\n limits:\n memory: 512Mi # Peak was 400Mi + 25%\n cpu: \\\"1.0\\\" # Peak was 0.8 + 25%\n requests:\n memory: 256Mi # Average was 250Mi\n cpu: \\\"0.5\\\" # Average was 0.4\n```\n\nKey principle: Don't guess, measure. Profile in staging, validate in production, iterate.\"\n\n### Q5: What's the difference between --cpus, --cpu-shares, and --cpuset-cpus?\n\n**Strong Answer:**\n\n\"These control different aspects of CPU allocation:\n\n**--cpus (Hard quota):**\n- Limits total CPU time\n- `--cpus=1.5` means container can use 150% of one core\n- Enforced regardless of host CPU availability\n- Container is throttled if it tries to use more\n- Use when: Predictable resource allocation, multi-tenant environments\n\n**--cpu-shares (Relative weight):**\n- Default is 1024\n- Only matters under contention\n- `--cpu-shares=512` gets half the CPU of a 1024 container\n- Can burst to full CPU if no contention\n- Use when: Batch jobs, where burst capability is valuable\n\n**--cpuset-cpus (Core pinning):**\n- Restricts to specific CPU cores\n- `--cpuset-cpus=0,1` only runs on cores 0 and 1\n- Good for: NUMA locality, cache optimization, isolating workloads\n- Use when: Latency-sensitive apps, NUMA-aware deployments\n\n**Combined example:**\n```bash\ndocker run \\\n --cpus=1.5 \\ # Max 1.5 CPU total\n --cpu-shares=2048 \\ # Higher priority than default\n --cpuset-cpus=0,1 \\ # Only use cores 0 and 1\n myapp\n```\n\nThis container:\n- Limited to 1.5 CPUs worth of time\n- Gets priority over default containers when competing\n- Only scheduled on cores 0 and 1\"\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Resource Limits Quick Reference\n\n| Resource | Flag | Enforcement |\n|----------|------|-------------|\n| Memory hard limit | `--memory` | OOM kill |\n| Memory soft limit | `--memory-reservation` | Reclaim under pressure |\n| CPU quota | `--cpus` | Throttling |\n| CPU weight | `--cpu-shares` | Proportional sharing |\n| CPU pinning | `--cpuset-cpus` | Core restriction |\n\n### Application Tuning Summary\n\n| Runtime | Memory Flag | Recommendation |\n|---------|-------------|----------------|\n| JVM | `-XX:MaxRAMPercentage=75.0` | Leave 25% for non-heap |\n| Node.js | `--max-old-space-size=N` | Set to ~75% of limit in MB |\n| Python | N/A | Generally automatic |\n\n### Golden Rules\n\n1. **Always set limits in production** - Prevent runaway containers\n2. **Leave headroom** - Heap ≠ container limit\n3. **Monitor throttling** - It causes latency, not just reduced throughput\n4. **Profile first** - Measure, don't guess\n5. **Make apps container-aware** - JVM, Node need explicit configuration\n\n---\n\n## 📋 Quick Reference\n\n### Common Commands\n\n```bash\n# Run with limits\ndocker run --memory=512m --cpus=1.0 myapp\n\n# Check OOM\ndocker inspect container --format '{{.State.OOMKilled}}'\n\n# Monitor resources\ndocker stats\n\n# Check cgroup limits\ncat /sys/fs/cgroup/docker//memory.max\ncat /sys/fs/cgroup/docker//cpu.max\n\n# Check throttling\ncat /sys/fs/cgroup/docker//cpu.stat\n```\n\n### JVM Flags\n\n```bash\n-XX:+UseContainerSupport # Respect container limits\n-XX:MaxRAMPercentage=75.0 # Heap as % of container\n-XX:InitialRAMPercentage=50.0 # Initial heap\n-XX:MaxMetaspaceSize=128m # Limit metaspace\n```\n\n### Exit Codes\n\n| Code | Meaning |\n|------|---------|\n| 137 | SIGKILL (OOM or docker kill) |\n| 143 | SIGTERM (graceful shutdown) |\n| 139 | SIGSEGV (segmentation fault) |\n\n---\n\n## 📅 Review Schedule\n\n| Day | Task | Time |\n|-----|------|------|\n| Day 1 | Review memory/CPU limit flags | 10 min |\n| Day 3 | Do Exercise 1 (observe OOM) | 15 min |\n| Day 7 | Configure JVM for container in real project | 20 min |\n| Day 14 | Profile and right-size a service | 30 min |\n| Day 30 | Audit all services for resource limits | 30 min |\n\n---\n\n## 📚 Series Navigation\n\n| Previous | Current | Next |\n|----------|---------|------|\n| [Part 8: Build Configuration](docker-part-8-build-configuration) | **Part 9: Resource Management** | [Part 10: Volumes & Storage](docker-part-10-volumes) |\n\n**Docker Compendium Series:**\n- [Part 0: How to Use This Series](docker-part-0-how-to-use-this-series)\n- [Part 1: Container Internals](docker-part-1-container-internals)\n- [Part 2: Image Anatomy](docker-part-2-image-anatomy)\n- [Part 3: Build Process Deep Dive](docker-part-3-build-process)\n- [Part 4: Networking Internals](docker-part-4-networking-internals)\n- [Part 5: Dockerfile Optimization Patterns](docker-part-5-dockerfile-optimization)\n- [Part 6: Multi-Stage Builds: Beyond Basics](docker-part-6-multi-stage-builds)\n- [Part 7: Base Image Selection & Security](docker-part-7-base-images)\n- [Part 8: ARG, ENV & Build-Time Configuration](docker-part-8-build-configuration)\n- **Part 9: Container Resource Management** ← You are here\n- [Part 10: Volume Patterns & Data Persistence](docker-part-10-volumes)\n","2026-06-10T07:19:25.471218Z","2026-06-10T07:19:25.471802Z",17,9,{"_33":57,"_99":59,"_100":101,"_102":103,"_104":-5,"_105":-5,"_106":-5,"_107":-5,"_108":-5},{"_33":95,"_73":61,"_37":96,"_76":97,"_98":-5},[122,123,124,125],{"_33":57,"_73":74,"_37":75,"_76":77,"_78":24},{"_33":92,"_73":93,"_37":94},{"_33":129,"_73":130,"_37":131},{"_33":89,"_73":90,"_37":91},{"_33":126,"_73":127,"_37":128},233,"Memory","memory",234,"Cpu","cpu"]