Container Internals: What's Really Running

📋 At a Glance

---

🎯 What You'll Learn

• Linux Namespaces - how Docker isolates processes, networks, and filesystems
• Control Groups (cgroups) - how resource limits actually work
• The container lifecycle - what happens from docker run to process exit
• OCI Runtime - the actual binary that creates containers
• Why this matters - debugging and optimization implications

---

🔥 Production Story: The Zombie Process Apocalypse

A team deployed a Python application processing video files. Each request spawned FFmpeg subprocesses. After a week in production, the container showed 1,247 processes - most were zombies (defunct processes).

• Container memory slowly grew
• Process table filled up
• Eventually: "fork: Resource temporarily unavailable"

This injects a minimal init process (tini) as PID 1 that properly reaps zombies. Without understanding PID namespaces and init responsibilities, this bug was "unexplainable."

---

🧠 Mental Model: Containers = Processes + Isolation

Forget "lightweight VMs." A container is:

┌────────────────────────────────────────────────────────────────────┐
│                         HOST KERNEL                                  │
│                    (shared by all containers)                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌──────────────────────┐    ┌──────────────────────┐               │
│  │     Container A      │    │     Container B      │               │
│  │                      │    │                      │               │
│  │  ┌────────────────┐  │    │  ┌────────────────┐  │               │
│  │  │   NAMESPACES   │  │    │  │   NAMESPACES   │  │   Isolation   │
│  │  │  ┌──────────┐  │  │    │  │  ┌──────────┐  │  │               │
│  │  │  │ PID: 1   │  │  │    │  │  │ PID: 1   │  │  │               │
│  │  │  │ NET: own │  │  │    │  │  │ NET: own │  │  │               │
│  │  │  │ MNT: own │  │  │    │  │  │ MNT: own │  │  │               │
│  │  │  │ UTS: own │  │  │    │  │  │ UTS: own │  │  │               │
│  │  │  └──────────┘  │  │    │  │  └──────────┘  │  │               │
│  │  └────────────────┘  │    │  └────────────────┘  │               │
│  │                      │    │                      │               │
│  │  ┌────────────────┐  │    │  ┌────────────────┐  │               │
│  │  │    CGROUPS     │  │    │  │    CGROUPS     │  │   Resource    │
│  │  │  mem: 512MB    │  │    │  │  mem: 1GB      │  │   Limits      │
│  │  │  cpu: 0.5      │  │    │  │  cpu: 2.0      │  │               │
│  │  └────────────────┘  │    │  └────────────────┘  │               │
│  │                      │    │                      │               │
│  │  ┌────────────────┐  │    │  ┌────────────────┐  │               │
│  │  │  FILESYSTEM    │  │    │  │  FILESYSTEM    │  │   Overlay     │
│  │  │  (overlay)     │  │    │  │  (overlay)     │  │   FS          │
│  │  └────────────────┘  │    │  └────────────────┘  │               │
│  └──────────────────────┘    └──────────────────────┘               │
│                                                                     │
│   Host PID 1234 ─────────────────── Host PID 5678                   │
│   (nginx in Container A)           (java in Container B)            │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

---

🔬 Deep Dive

Linux Namespaces: The Isolation Mechanism

Namespaces give each container its own view of system resources. There are 8 namespace types:

PID Namespace Deep Dive

BASH(10 lines)
Code
Loading syntax highlighter...

• Host PID: 1234 (real)
• Container PID: 1 (virtualized)

1. Reap orphaned child processes (zombie cleanup)
2. Forward signals to children
3. Handle SIGTERM gracefully for container shutdown

Most applications aren't designed to be PID 1. Solutions:

• docker run --init - injects tini as PID 1
• Use dumb-init or tini in your Dockerfile
• Properly handle signals in your app

Network Namespace Deep Dive

Each container gets its own:

• Network interfaces (eth0, lo)
• Routing tables
• iptables rules
• Port bindings

BASH(11 lines)
Code
Loading syntax highlighter...

┌─────────────────────────────────────────────────────┐
│                      HOST                           │
│                                                     │
│  ┌────────────┐         ┌────────────┐              │
│  │ Container  │         │  docker0   │              │
│  │            │         │  bridge    │              │
│  │  eth0 ─────┼─ veth ──┼─ 172.17.0.1│── eth0 ──────┼──► Internet
│  │ 172.17.0.2 │  pair   │            │              │
│  │            │         │            │              │
│  └────────────┘         └────────────┘              │
│                                                     │
└─────────────────────────────────────────────────────┘

The veth pair is like a virtual network cable connecting the container's namespace to the host's docker0 bridge.

Mount Namespace Deep Dive

Containers see a different filesystem than the host:

BASH(7 lines)
Code
Loading syntax highlighter...

BASH(2 lines)
Code
Loading syntax highlighter...

Control Groups (cgroups): Resource Limits

Namespaces provide isolation. Cgroups provide resource limits.

cgroup v1 vs v2

BASH(6 lines)
Code
Loading syntax highlighter...

• Single unified hierarchy
• Cleaner resource distribution
• Better pressure stall information (PSI)
• Required for some K8s features

Memory Limits in Practice

BASH(6 lines)
Code
Loading syntax highlighter...

┌─────────────────────────────────────────────────────┐
│              Container Memory Usage                 │
│                                                     │
│  ┌─────────────────────────────────────────────────┐│
│  │                                                 ││
│  │  ████████████████████░░░░░░░░░░░  450MB         ││
│  │                                                 ││
│  │  ─────────────────────────────── 512MB limit    ││
│  └─────────────────────────────────────────────────┘│
│                                                     │
│  If exceeds limit:                                  │
│  1. Kernel tries to reclaim memory                  │
│  2. If can't: OOM killer activates                  │
│  3. Container's main process killed (SIGKILL)       │
│  4. Exit code: 137 (128 + 9)                        │
└─────────────────────────────────────────────────────┘

CPU Limits in Practice

Two main mechanisms:

BASH(8 lines)
Code
Loading syntax highlighter...

What Happens During docker run

┌─────────────────────────────────────────────────────────────────┐
│                    docker run -d nginx                          │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 1. Docker CLI → Docker Daemon (dockerd)                         │
│    - Parse arguments                                            │
│    - Validate image exists (or pull)                            │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 2. Docker Daemon → containerd                                   │
│    - Create container metadata                                  │
│    - Prepare rootfs (overlay mount)                             │
│    - Generate OCI runtime spec (config.json)                    │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 3. containerd → containerd-shim → runc                          │
│    - shim: Keeps container alive if containerd restarts         │
│    - runc: Actually creates the container                       │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 4. runc creates container:                                      │
│    a. Create namespaces (clone syscall with namespace flags)    │
│    b. Set up cgroups                                            │
│    c. Configure seccomp filters                                 │
│    d. Set capabilities                                          │
│    e. Pivot root to container filesystem                        │
│    f. Execute entrypoint (nginx)                                │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 5. Container running!                                           │
│    - Process running in isolated namespaces                     │
│    - Resource limits enforced by cgroups                        │
│    - containerd-shim monitors for exit                          │
└─────────────────────────────────────────────────────────────────┘

The OCI Runtime Specification

OCI (Open Container Initiative) defines standards for containers. The runtime spec defines:

1. config.json - Container configuration

JSON(25 lines)
Code
Loading syntax highlighter...

1. rootfs/ - The container's filesystem

This standardization means you can use different runtimes:

• runc - Default, reference implementation
• crun - Faster, written in C
• gVisor (runsc) - Sandboxed, intercepts syscalls
• Kata Containers - VM-level isolation

Verifying Isolation from the Host

You can peek inside namespaces from the host:

BASH(17 lines)
Code
Loading syntax highlighter...

---

⚠️ Common Mistakes

Mistake 1: Treating Containers Like VMs

BASH(10 lines)
Code
Loading syntax highlighter...

Mistake 2: Ignoring PID 1 Responsibilities

BASH(13 lines)
Code
Loading syntax highlighter...

Mistake 3: Assuming Container Isolation = Security

BASH(14 lines)
Code
Loading syntax highlighter...

---

🐛 Debug This: The Invisible Process

BASH(12 lines)
Code
Loading syntax highlighter...

---

BASH(10 lines)
Code
Loading syntax highlighter...

---

💻 Exercises

Exercise 1: Explore Your Container's Namespaces

⭐ Difficulty: Easy | ⏱️ Time: 15 minutes

BASH(13 lines)
Code
Loading syntax highlighter...

---

Exercise 2: Watch OOM Killer in Action

⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes

BASH(23 lines)
Code
Loading syntax highlighter...

---

Exercise 3: Create Container "By Hand" with unshare

⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 30 minutes

Create a minimal container without Docker using Linux primitives:

BASH(19 lines)
Code
Loading syntax highlighter...

---

Exercise 4: Measure Namespace Overhead

⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 25 minutes

BASH(18 lines)
Code
Loading syntax highlighter...

---

Exercise 5: Debug Zombie Processes

⭐⭐⭐⭐ Difficulty: Expert | ⏱️ Time: 30 minutes

BASH(33 lines)
Code
Loading syntax highlighter...

---

🎤 Senior-Level Interview Questions

Q1: What's the difference between a container and a VM?

"The fundamental difference is the isolation level. VMs virtualize hardware - each VM runs its own kernel, booting a full OS. The hypervisor manages hardware sharing.

Containers virtualize the OS - they share the host kernel but use Linux namespaces for isolation and cgroups for resource limits. This means:

1. Startup: Containers start in milliseconds (just a process), VMs in seconds/minutes (full boot)
2. Overhead: Containers use MBs of memory overhead, VMs use GBs
3. Isolation: VMs have stronger isolation (separate kernel). A kernel vulnerability affects all containers but only one VM
4. Density: You can run 100s of containers on a host, maybe 10s of VMs

For most workloads, containers are better for efficiency. VMs are better when you need strong multi-tenant isolation or different operating systems."

Q2: A container process shows as PID 1 inside but PID 5000 on the host. Explain.

Inside this namespace, PIDs start from 1 - the first process becomes PID 1. But the kernel still tracks the 'real' PID in the root namespace.

The process has two PIDs simultaneously:

• Namespace PID: 1 (what the container sees)
• Host PID: 5000 (what the kernel uses)

Q3: How do you debug a container that exits immediately with code 137?

"Exit code 137 is 128 + 9, meaning the process received SIGKILL. This is almost always the OOM killer.

My debugging steps:

1. Confirm OOM: docker inspect shows OOMKilled: true
2. Check memory limit: Was it set too low? docker inspect --format '{{.HostConfig.Memory}}'
3. Check actual usage: docker stats before it dies, or kernel logs: dmesg | grep -i 'killed process'
4. Analyze the app: Is it a memory leak, or does it legitimately need more?
5. Fix options:
   • Increase limit if app needs it
   • Fix memory leak if there is one
   • For JVM: ensure -XX:+UseContainerSupport and set -XX:MaxRAMPercentage
   • Add memory monitoring/alerting before OOM

The key insight: OOM kills are silent from the container's perspective. No logs, no graceful shutdown, just SIGKILL."

Q4: Why might a process inside a container not see other processes also inside that container?

"If a container is running multiple processes and one can't see the others, there are a few possibilities:

1. Different PID namespaces: If a process was started with additional isolation, it might have its own PID namespace. Docker --init does this correctly, but custom setups might not.
2. Proc filesystem not mounted: If /proc isn't mounted (or mounted incorrectly), ps won't show processes. This happens in minimal containers.
3. Incorrect /proc mount: If /proc was mounted from outside the namespace, it shows the wrong namespace's processes.
4. Process actually died: Race condition - process started but exited before ps ran.

Q5: Explain what happens between docker run nginx and nginx actually serving requests.

"The full sequence involves multiple components:

1. Docker CLI parses the command, communicates with Docker daemon via REST API over Unix socket
2. Docker daemon (dockerd):
   • Checks image exists locally, pulls if not
   • Creates container metadata
   • Sets up networking (creates veth pair, connects to bridge)
   • Prepares storage (overlay mount)
3. Containerd:
   • Receives request from dockerd
   • Creates OCI runtime bundle (config.json + rootfs)
   • Spawns containerd-shim
4. Containerd-shim:
   • Stays running to monitor container
   • Survives containerd restarts
   • Calls runc
5. Runc (OCI runtime):
   • Creates namespaces (clone syscall)
   • Sets up cgroups
   • Applies seccomp filters
   • Sets capabilities
   • Performs pivot_root to container's filesystem
   • Execs nginx
6. Nginx starts as PID 1 in container (PID N on host)
7. Network path:
   • Nginx binds to port 80 inside container
   • Docker sets up iptables DNAT rule
   • External traffic to host:8080 → container:80

Total time: ~100-500ms for a cached image."

---

📝 Summary & Key Takeaways

Core Concepts

The Container Equation

Container = Linux Process
          + Namespaces (isolation)
          + Cgroups (resource limits)
          + Overlay filesystem (image)

What You Can Do Now

1. Debug namespace issues: Use nsenter and lsns to inspect containers
2. Understand resource limits: Know why OOM happens (exit code 137)
3. Avoid zombie processes: Use --init or proper signal handling
4. Explain containers technically: Move beyond "lightweight VMs" explanation

---

📋 Quick Reference

Namespace Types

Key Commands

BASH(14 lines)
Code
Loading syntax highlighter...

Exit Codes

---

📅 Review Schedule

---

📚 Series Navigation

• Part 0: How to Use This Series
• Part 1: Container Internals ← You are here
• Part 2: Image Anatomy
• Part 3: Build Process Deep Dive
• Part 4: Networking Internals