[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6},"actionData","errors","routes/blog-tag",{"_7":8,"_9":10},"tag",{"_31":84,"_72":85,"_35":86},"articles",{"_11":12,"_13":14,"_15":16,"_17":16,"_18":19,"_20":21,"_22":23,"_24":23,"_25":16,"_26":27},"content",[30],"pageable",{"_28":21,"_29":19},"totalElements",1,"totalPages","size",10,"number",0,"first",true,"last","numberOfElements","empty",false,"page_number","page_size",{"_31":32,"_33":34,"_35":36,"_37":38,"_11":39,"_40":-5,"_41":23,"_42":27,"_43":44,"_45":46,"_47":48,"_49":50,"_51":52,"_53":52,"_54":55,"_56":57,"_58":59,"_60":52,"_61":55,"_62":63,"_64":65,"_66":67,"_68":69,"_70":71},"id",52,"title","Image Anatomy: Layers, Manifests & Registry","slug","docker-part-2-image-anatomy","excerpt","Deep dive into image layers, content-addressable storage, and multi-arch","# Docker Compendium Part 2: Image Anatomy - Layers, Manifests & Registry\n\nYou `docker pull` an image and it downloads in layers. But what are layers really? How does Docker know which parts to download? This article dissects Docker images - from the overlay filesystem to multi-architecture manifests to content-addressable storage.\n\n## 📋 At a Glance\n\n| Aspect | Details |\n|--------|---------|\n| Topic | Image layers, content-addressable storage, registries, multi-arch |\n| Complexity | Advanced |\n| Prerequisites | Part 1 (Container Internals), basic Docker usage |\n| Key Insight | Images are just tarballs with metadata, layers are deduplicated filesystem diffs |\n| Time to Master | 3-4 hours |\n\n---\n\n## 🎯 What You'll Learn\n\n- **Layer mechanics** - how images are built from filesystem diffs\n- **Content-addressable storage** - why layer hashes matter\n- **Image manifests** - the metadata that ties layers together\n- **Multi-architecture images** - how one tag serves ARM and AMD64\n- **Registry protocol** - what happens during push/pull\n\n---\n\n## 🔥 Production Story: 50GB of Dangling Layers\n\nA CI server's disk filled up every week. The team blamed \"too many builds\" and added cron jobs to clean up. But the real problem was worse.\n\n**Investigation revealed:**\n- 847 dangling images consuming 47GB\n- Each build pulled base image, built, but never cleaned intermediates\n- Layer deduplication wasn't working - same base layers stored multiple times\n\n**Root cause:** They were using `docker build` without `--rm` and never running `docker image prune`. But deeper: they didn't understand that every build command creates a layer, and layers persist until explicitly removed.\n\n**The fix:**\n```bash\n# After each build\ndocker image prune -f\n\n# Weekly cleanup\ndocker system prune -a --volumes\n\n# Better: Use BuildKit with proper garbage collection\nDOCKER_BUILDKIT=1 docker build ...\n```\n\n**What they learned:** Image storage isn't magic. Every layer is a tarball on disk. Understanding layer structure means understanding disk usage.\n\n---\n\n## 🧠 Mental Model: Images as Layer Stacks\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ IMAGE STRUCTURE │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ Image: nginx:1.25 │\n│ │\n│ ┌────────────────────────────────────────────────────────────┐ │\n│ │ Manifest (JSON) │ │\n│ │ - Config blob reference │ │\n│ │ - Layer references (in order) │ │\n│ │ - Media types │ │\n│ └────────────────────────────────────────────────────────────┘ │\n│ │ │\n│ ▼ │\n│ ┌────────────────────────────────────────────────────────────┐ │\n│ │ Config (JSON) │ │\n│ │ - Environment variables │ │\n│ │ - CMD, ENTRYPOINT │ │\n│ │ - Exposed ports │ │\n│ │ - History (build steps) │ │\n│ └────────────────────────────────────────────────────────────┘ │\n│ │ │\n│ ▼ │\n│ ┌────────────────────────────────────────────────────────────┐ │\n│ │ LAYER 4: sha256:a1b2... (12MB) ← nginx config │ │\n│ │ Changes: /etc/nginx/nginx.conf, /usr/share/nginx/html/ │ │\n│ ├────────────────────────────────────────────────────────────┤ │\n│ │ LAYER 3: sha256:c3d4... (25MB) ← nginx binary │ │\n│ │ Changes: /usr/sbin/nginx, /usr/lib/nginx/ │ │\n│ ├────────────────────────────────────────────────────────────┤ │\n│ │ LAYER 2: sha256:e5f6... (45MB) ← apt packages │ │\n│ │ Changes: /usr/bin/*, /usr/lib/* │ │\n│ ├────────────────────────────────────────────────────────────┤ │\n│ │ LAYER 1: sha256:7890... (80MB) ← debian:bookworm-slim │ │\n│ │ Changes: /bin/*, /lib/*, /etc/* │ │\n│ └────────────────────────────────────────────────────────────┘ │\n│ │\n│ Total size: 162MB (but layers shared with other images!) │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**Key insight:** Layers are filesystem diffs, not complete snapshots. Each layer contains only what changed from the layer below.\n\n---\n\n## 🔬 Deep Dive\n\n### How Layers Work: Union Filesystem\n\nDocker uses **overlay filesystem** (overlayfs) to stack layers:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ CONTAINER FILESYSTEM VIEW │\n│ │\n│ What container sees: /bin /etc /usr /var /app ... │\n│ │\n├─────────────────────────────────────────────────────────────────┤\n│ │\n│ ┌──────────────┐ Container layer (read-write) │\n│ │ UPPER │ - Container's changes go here │\n│ │ (rw) │ - Created fresh for each container │\n│ └──────────────┘ │\n│ │ │\n│ │ overlay merge │\n│ ▼ │\n│ ┌──────────────┐ │\n│ │ LAYER 4 │ Image layers (read-only) │\n│ │ (ro) │ │\n│ ├──────────────┤ │\n│ │ LAYER 3 │ │\n│ │ (ro) │ │\n│ ├──────────────┤ │\n│ │ LAYER 2 │ │\n│ │ (ro) │ │\n│ ├──────────────┤ │\n│ │ LAYER 1 │ │\n│ │ (ro) │ │\n│ └──────────────┘ │\n│ │\n└─────────────────────────────────────────────────────────────────┘\n\nFile resolution (top-down):\n1. Check UPPER - if file exists, use it\n2. Check LAYER 4 - if file exists, use it\n3. Check LAYER 3...\n4. Continue down until found\n\nWrite operations:\n- New file → write to UPPER\n- Modify existing → copy to UPPER, modify (copy-on-write)\n- Delete → create \"whiteout\" marker in UPPER\n```\n\n**See it in action:**\n\n```bash\n# Start a container\ndocker run -d --name overlay-demo nginx\n\n# Find the overlay mount\ndocker inspect overlay-demo --format '{{.GraphDriver.Data.MergedDir}}'\n# /var/lib/docker/overlay2/abc123.../merged\n\n# See the layers\ndocker inspect overlay-demo --format '{{.GraphDriver.Data.LowerDir}}'\n# /var/lib/docker/overlay2/layer1:/var/lib/docker/overlay2/layer2:...\n\n# See container's writeable layer\ndocker inspect overlay-demo --format '{{.GraphDriver.Data.UpperDir}}'\n# /var/lib/docker/overlay2/abc123.../diff\n```\n\n### Content-Addressable Storage\n\nEvery piece of image data is identified by its **SHA256 hash**:\n\n```bash\n# Pull an image and see layer digests\ndocker pull nginx:1.25\n# 1.25: Pulling from library/nginx\n# a2abf6c4d29d: Pull complete ← Layer digest (short form)\n# a9edb18cadd1: Pull complete\n# 589b7251471a: Pull complete\n# ...\n\n# Full digest format\nsha256:a2abf6c4d29d3c8b3b4c5e6f7a8b9c0d1e2f3a4b5c6d7e8f9...\n\n# This hash is the SHA256 of the layer's tarball\n# If ANY byte changes, the hash changes completely\n```\n\n**Why content-addressable storage matters:**\n\n1. **Deduplication:** Same layer content = same hash = store once\n2. **Integrity:** Downloaded data must match expected hash\n3. **Caching:** Already have this hash? Don't download again\n4. **Immutability:** Can't modify a layer without changing its hash\n\n```bash\n# See local image storage\nls /var/lib/docker/image/overlay2/layerdb/sha256/\n\n# Each directory is a layer, named by its chain ID\n# Chain ID = SHA256(parent chain ID + layer diff ID)\n```\n\n### Image Manifest Structure\n\nThe manifest ties everything together:\n\n```bash\n# Get image manifest\ndocker manifest inspect nginx:1.25\n```\n\n```json\n{\n \"schemaVersion\": 2,\n \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n \"config\": {\n \"mediaType\": \"application/vnd.docker.container.image.v1+json\",\n \"size\": 7023,\n \"digest\": \"sha256:abc123...\"\n },\n \"layers\": [\n {\n \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n \"size\": 31456789,\n \"digest\": \"sha256:a2abf6c4d29d...\"\n },\n {\n \"mediaType\": \"application/vnd.docker.image.rootfs.diff.tar.gzip\",\n \"size\": 25123456,\n \"digest\": \"sha256:a9edb18cadd1...\"\n }\n ]\n}\n```\n\n**Manifest components:**\n\n| Field | Purpose |\n|-------|---------|\n| `config.digest` | Points to image config JSON |\n| `layers[].digest` | SHA256 of compressed layer tarball |\n| `layers[].size` | Size in bytes (for progress display) |\n| `mediaType` | Format identifier |\n\n### Image Config Deep Dive\n\n```bash\n# Extract image config\ndocker image inspect nginx:1.25 --format '{{.Config}}'\n\n# Or get the full config blob\ndocker save nginx:1.25 | tar -xO manifest.json | jq\ndocker save nginx:1.25 | tar -xO abc123.json # config blob\n```\n\n```json\n{\n \"architecture\": \"amd64\",\n \"os\": \"linux\",\n \"config\": {\n \"Hostname\": \"\",\n \"Env\": [\n \"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\n \"NGINX_VERSION=1.25.3\"\n ],\n \"Cmd\": [\"nginx\", \"-g\", \"daemon off;\"],\n \"ExposedPorts\": { \"80/tcp\": {} },\n \"StopSignal\": \"SIGQUIT\"\n },\n \"rootfs\": {\n \"type\": \"layers\",\n \"diff_ids\": [\n \"sha256:a2abf6c4d29d...\",\n \"sha256:a9edb18cadd1...\"\n ]\n },\n \"history\": [\n {\n \"created\": \"2023-11-21T00:00:00.000000000Z\",\n \"created_by\": \"/bin/sh -c #(nop) ADD file:... in / \"\n },\n {\n \"created\": \"2023-11-21T00:00:01.000000000Z\",\n \"created_by\": \"/bin/sh -c apt-get update && apt-get install...\"\n }\n ]\n}\n```\n\n**History section reveals build steps** - useful for understanding how an image was built.\n\n### Multi-Architecture Images (Manifest Lists)\n\nOne tag can serve multiple architectures:\n\n```bash\n# Inspect multi-arch image\ndocker manifest inspect --verbose nginx:1.25\n```\n\n```json\n{\n \"schemaVersion\": 2,\n \"mediaType\": \"application/vnd.docker.distribution.manifest.list.v2+json\",\n \"manifests\": [\n {\n \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n \"digest\": \"sha256:amd64manifest...\",\n \"platform\": {\n \"architecture\": \"amd64\",\n \"os\": \"linux\"\n }\n },\n {\n \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n \"digest\": \"sha256:arm64manifest...\",\n \"platform\": {\n \"architecture\": \"arm64\",\n \"os\": \"linux\",\n \"variant\": \"v8\"\n }\n },\n {\n \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n \"digest\": \"sha256:armv7manifest...\",\n \"platform\": {\n \"architecture\": \"arm\",\n \"os\": \"linux\",\n \"variant\": \"v7\"\n }\n }\n ]\n}\n```\n\n**How it works:**\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ docker pull nginx:1.25 │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ Registry Response │\n│ │\n│ Manifest List (nginx:1.25) │\n│ ├─ amd64/linux → sha256:amd64manifest │\n│ ├─ arm64/linux → sha256:arm64manifest │\n│ └─ arm/v7/linux → sha256:armv7manifest │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n │ Client selects based on\n │ local architecture\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ On M1 Mac: Pull sha256:arm64manifest │\n│ On x86 PC: Pull sha256:amd64manifest │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**Creating multi-arch images:**\n\n```bash\n# Build for multiple platforms\ndocker buildx build \\\n --platform linux/amd64,linux/arm64 \\\n --tag myapp:1.0 \\\n --push .\n\n# Or create manifest list manually\ndocker manifest create myapp:1.0 \\\n myapp:1.0-amd64 \\\n myapp:1.0-arm64\ndocker manifest push myapp:1.0\n```\n\n### Registry Protocol (OCI Distribution)\n\nWhat happens during `docker pull`:\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ docker pull nginx:1.25 │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ Step 1: Resolve tag to manifest │\n│ │\n│ GET /v2/library/nginx/manifests/1.25 │\n│ Accept: application/vnd.docker.distribution.manifest.v2+json │\n│ │\n│ Response: Manifest JSON with config + layer digests │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ Step 2: Download config blob │\n│ │\n│ GET /v2/library/nginx/blobs/sha256:configdigest │\n│ │\n│ Response: Image config JSON │\n└──────────────────────────┬──────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────┐\n│ Step 3: Download layers (parallel, if not cached) │\n│ │\n│ For each layer: │\n│ - Check if sha256:layerdigest exists locally │\n│ - If not: GET /v2/library/nginx/blobs/sha256:layerdigest │\n│ - Verify downloaded hash matches │\n│ - Decompress and store │\n└─────────────────────────────────────────────────────────────────┘\n```\n\n**What happens during `docker push`:**\n\n```bash\n# Push sequence\n1. Check which layers registry already has (HEAD requests)\n2. Upload missing layers (POST + PUT with resumable upload)\n3. Upload config blob\n4. Upload manifest (ties everything together)\n```\n\n### Analyzing Image Layers\n\n```bash\n# See layer sizes and commands\ndocker history nginx:1.25\n# IMAGE CREATED CREATED BY SIZE\n# sha256:abc1 2 weeks ago CMD [\"nginx\" \"-g\" \"daemon off;\"] 0B\n# sha256:abc2 2 weeks ago EXPOSE map[80/tcp:{}] 0B\n# sha256:abc3 2 weeks ago STOPSIGNAL SIGQUIT 0B\n# sha256:abc4 2 weeks ago RUN /bin/sh -c set -x && addgroup... 63.1MB\n# ...\n\n# Detailed layer analysis with dive\ndive nginx:1.25\n# Interactive TUI showing:\n# - Layer contents\n# - File changes per layer\n# - Wasted space\n# - Efficiency score\n```\n\n### Layer Caching Mechanics\n\nWhy does rebuild sometimes use cache, sometimes not?\n\n```dockerfile\n# Dockerfile\nFROM node:20\nWORKDIR /app\nCOPY package.json .\nRUN npm install\nCOPY . .\nCMD [\"node\", \"index.js\"]\n```\n\n**Cache decision for each instruction:**\n\n| Instruction | Cache Key |\n|-------------|-----------|\n| `FROM` | Base image digest |\n| `WORKDIR` | Previous layer + instruction |\n| `COPY` | Previous layer + file content hashes |\n| `RUN` | Previous layer + command string |\n| `CMD` | Previous layer + command |\n\n**Cache invalidation cascade:**\n\n```\nIf package.json changes:\n ├─ COPY package.json . → MISS (file changed)\n ├─ RUN npm install → MISS (parent changed)\n ├─ COPY . . → MISS (parent changed)\n └─ All subsequent layers rebuild\n\nIf only src/index.js changes:\n ├─ COPY package.json . → HIT\n ├─ RUN npm install → HIT\n ├─ COPY . . → MISS (file changed)\n └─ Only this and following rebuild\n```\n\nThis is why **order matters** in Dockerfiles (covered in Part 5).\n\n### Where Images Live on Disk\n\n```bash\n# Docker root directory\nls /var/lib/docker/\n\n# Image metadata\nls /var/lib/docker/image/overlay2/\n# ├── distribution/ # Tag → digest mappings\n# ├── imagedb/ # Image configs\n# │ └── content/sha256/ # Actual config JSONs\n# └── layerdb/ # Layer metadata\n# ├── sha256/ # By diff ID\n# └── mounts/ # Container layer mappings\n\n# Actual layer data (tarballs extracted)\nls /var/lib/docker/overlay2/\n# Each directory is a layer's extracted contents\n# Named by random ID (not content hash)\n```\n\n**Cleanup commands:**\n\n```bash\n# Remove unused images\ndocker image prune\n\n# Remove ALL unused images (not just dangling)\ndocker image prune -a\n\n# See disk usage breakdown\ndocker system df\ndocker system df -v # Verbose\n\n# Nuclear option - remove everything unused\ndocker system prune -a --volumes\n```\n\n---\n\n## ⚠️ Common Mistakes\n\n### Mistake 1: Not Understanding Layer Accumulation\n\n```dockerfile\n# ❌ Creates large layers that can't be cleaned\nFROM ubuntu:22.04\nRUN apt-get update\nRUN apt-get install -y build-essential\nRUN apt-get clean\n# The 'clean' doesn't reduce image size!\n# apt-get update cache is in layer 2\n# packages are in layer 3\n# 'clean' is in layer 4 (removes nothing from previous layers)\n\n# ✅ Single layer, cleanup in same RUN\nFROM ubuntu:22.04\nRUN apt-get update && \\\n apt-get install -y build-essential && \\\n apt-get clean && \\\n rm -rf /var/lib/apt/lists/*\n# One layer with only final state\n```\n\n### Mistake 2: Ignoring Image Provenance\n\n```bash\n# ❌ Trusting any image\ndocker pull someguy/nginx-optimized\n# Who is someguy? What's in this image?\n\n# ✅ Verify image source\n# Use official images\ndocker pull nginx\n\n# Or verify signatures (Docker Content Trust)\nexport DOCKER_CONTENT_TRUST=1\ndocker pull nginx # Will verify signatures\n\n# Scan for vulnerabilities\ndocker scout cves nginx:1.25\n```\n\n### Mistake 3: Using :latest in Production\n\n```bash\n# ❌ Mutable tag\ndocker pull myapp:latest\n# Tomorrow this might be completely different!\n\n# ✅ Immutable reference\ndocker pull myapp@sha256:abc123def456...\n# This is guaranteed to always be the same image\n\n# Or use semantic versioning\ndocker pull myapp:1.2.3\n# Pin to specific version\n```\n\n---\n\n## 🐛 Debug This: The Missing Layer Mystery\n\nA developer reports: \"I pushed my image, then pulled on another machine, but it's different. Some files are missing!\"\n\n```bash\n# Build and push\ndocker build -t myregistry/app:1.0 .\ndocker push myregistry/app:1.0\n\n# On another machine\ndocker pull myregistry/app:1.0\ndocker run myregistry/app:1.0 ls /app/config\n# ls: cannot access '/app/config': No such file or directory\n\n# But on build machine\ndocker run myregistry/app:1.0 ls /app/config\n# config.json\n# secrets.env\n```\n\n**Why are files missing on the second machine?**\n\n---\n\n**✅ Solution:**\n\nThe issue is likely **layer caching with different base layers**.\n\n**Scenario 1: Old cached layers**\n\nThe second machine had an old version of a base layer cached:\n\n```bash\n# Machine 2 had old layer cached\n# docker pull downloads manifest\n# Sees layer sha256:abc123 - \"already have it!\"\n# But local layer is from older build with different content\n\n# Fix: Force fresh pull\ndocker pull --pull-policy always myregistry/app:1.0\n# Or\ndocker image rm myregistry/app:1.0\ndocker pull myregistry/app:1.0\n```\n\n**Scenario 2: Build cache leak**\n\nThe build machine used cached layers from a previous build that weren't pushed:\n\n```bash\n# Build machine had intermediate layers cached\n# Those layers weren't pushed to registry\n# Final image reference points to layers that don't exist in registry\n\n# Fix: Build with --no-cache or ensure all layers pushed\ndocker build --no-cache -t myregistry/app:1.0 .\ndocker push myregistry/app:1.0\n```\n\n**Scenario 3: Multi-stage build artifact**\n\nFiles were in build stage but not copied to final stage:\n\n```dockerfile\n# Dockerfile\nFROM node:20 AS builder\nCOPY . /app\nRUN npm run build\n\nFROM node:20-slim\nWORKDIR /app\nCOPY --from=builder /app/dist ./dist\n# Forgot to copy /app/config!\n```\n\n**Debug steps:**\n\n```bash\n# Compare image digests\ndocker inspect myregistry/app:1.0 --format '{{.Id}}'\n# Run on both machines - should match\n\n# Compare layer digests\ndocker inspect myregistry/app:1.0 --format '{{.RootFS.Layers}}'\n\n# Export and compare\ndocker save myregistry/app:1.0 > image.tar\ntar -tvf image.tar # List contents\n```\n\n---\n\n## 💻 Exercises\n\n### Exercise 1: Dissect an Image\n⭐ Difficulty: Easy | ⏱️ Time: 15 minutes\n\n```bash\n# 1. Pull a small image\ndocker pull alpine:3.18\n\n# 2. View its layers\ndocker history alpine:3.18\n\n# 3. Export and explore\ndocker save alpine:3.18 -o alpine.tar\nmkdir alpine-extracted\ntar -xf alpine.tar -C alpine-extracted\nls alpine-extracted/\n\n# 4. Find and read the manifest\ncat alpine-extracted/manifest.json | jq\n\n# 5. Find and read the config\n# (filename from manifest.json)\ncat alpine-extracted/.json | jq\n\n# Questions:\n# - How many layers does Alpine have?\n# - What's the total uncompressed size?\n# - What commands created each layer?\n```\n\n---\n\n### Exercise 2: Measure Layer Deduplication\n⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes\n\n```bash\n# 1. Check current disk usage\ndocker system df\n\n# 2. Pull images sharing base layers\ndocker pull node:20\ndocker pull node:20-slim\ndocker pull node:20-alpine\n\n# 3. Check disk usage again\ndocker system df\n\n# 4. Calculate deduplication savings\n# Compare actual disk vs sum of image sizes\n\n# 5. Verify shared layers\ndocker inspect node:20 --format '{{.RootFS.Layers}}' > node20-layers.txt\ndocker inspect node:20-slim --format '{{.RootFS.Layers}}' > node20slim-layers.txt\ndiff node20-layers.txt node20slim-layers.txt\n# Or use: comm -12 to find common layers\n```\n\n---\n\n### Exercise 3: Build Multi-Architecture Image\n⭐⭐ Difficulty: Medium | ⏱️ Time: 25 minutes\n\n```bash\n# 1. Enable buildx\ndocker buildx create --use --name multiarch\n\n# 2. Create simple Dockerfile\ncat > Dockerfile << 'EOF'\nFROM alpine:3.18\nRUN apk add --no-cache curl\nCMD [\"curl\", \"--version\"]\nEOF\n\n# 3. Build for multiple architectures\ndocker buildx build \\\n --platform linux/amd64,linux/arm64 \\\n --tag myapp:multiarch \\\n --load .\n# Note: --load only works with single platform\n# Use --push with registry for true multi-arch\n\n# 4. Inspect the manifest list\ndocker manifest inspect myapp:multiarch\n\n# 5. Test on different architecture (if available)\n# On ARM Mac:\ndocker run --platform linux/amd64 myapp:multiarch\ndocker run --platform linux/arm64 myapp:multiarch\n```\n\n---\n\n### Exercise 4: Registry Protocol Deep Dive\n⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 30 minutes\n\n```bash\n# 1. Start local registry\ndocker run -d -p 5000:5000 --name registry registry:2\n\n# 2. Tag and push an image\ndocker tag alpine:3.18 localhost:5000/myalpine:1.0\ndocker push localhost:5000/myalpine:1.0\n\n# 3. Query registry API directly\n# Get catalog\ncurl http://localhost:5000/v2/_catalog\n\n# Get tags\ncurl http://localhost:5000/v2/myalpine/tags/list\n\n# Get manifest\ncurl -H \"Accept: application/vnd.docker.distribution.manifest.v2+json\" \\\n http://localhost:5000/v2/myalpine/manifests/1.0 | jq\n\n# 4. Download a layer blob directly\nDIGEST=$(curl -s -H \"Accept: application/vnd.docker.distribution.manifest.v2+json\" \\\n http://localhost:5000/v2/myalpine/manifests/1.0 | jq -r '.layers[0].digest')\n\ncurl http://localhost:5000/v2/myalpine/blobs/$DIGEST -o layer.tar.gz\n\n# 5. Inspect the layer\ntar -tzf layer.tar.gz | head -20\n```\n\n---\n\n### Exercise 5: Analyze Image Efficiency\n⭐⭐⭐⭐ Difficulty: Expert | ⏱️ Time: 30 minutes\n\n```bash\n# 1. Install dive (image analysis tool)\n# Mac: brew install dive\n# Linux: https://github.com/wagoodman/dive/releases\n\n# 2. Analyze an inefficient image\ncat > Dockerfile.inefficient << 'EOF'\nFROM ubuntu:22.04\nRUN apt-get update\nRUN apt-get install -y curl\nRUN apt-get install -y wget\nRUN apt-get install -y vim\nRUN apt-get clean\nRUN rm -rf /var/lib/apt/lists/*\nEOF\n\ndocker build -f Dockerfile.inefficient -t inefficient:1.0 .\ndive inefficient:1.0\n\n# 3. Create optimized version\ncat > Dockerfile.efficient << 'EOF'\nFROM ubuntu:22.04\nRUN apt-get update && \\\n apt-get install -y --no-install-recommends \\\n curl \\\n wget \\\n vim && \\\n apt-get clean && \\\n rm -rf /var/lib/apt/lists/*\nEOF\n\ndocker build -f Dockerfile.efficient -t efficient:1.0 .\ndive efficient:1.0\n\n# 4. Compare\ndocker images | grep -E 'efficient|inefficient'\n\n# Questions:\n# - What's the size difference?\n# - What efficiency score does dive show?\n# - Where is space wasted in the inefficient version?\n```\n\n---\n\n## 🎤 Senior-Level Interview Questions\n\n### Q1: Explain the difference between image ID, digest, and tag.\n\n**Strong Answer:**\n\n\"These are three different ways to reference images:\n\n**Tag** is a human-readable name like `nginx:1.25`. It's mutable - pushing a new image with the same tag overwrites the reference. Never use `:latest` in production because it can change.\n\n**Digest** is the SHA256 hash of the image manifest, like `sha256:abc123...`. It's immutable - this exact hash always refers to exactly this image. Format: `nginx@sha256:abc123...`\n\n**Image ID** is the SHA256 hash of the image config JSON (not the manifest). It's what you see in `docker images`. Two images with different tags can have the same ID if they're identical.\n\nIn practice:\n- Use tags for development convenience\n- Use digests for production deployments (immutability)\n- Image IDs are mainly for local identification\n\nThe relationship: Tag → Manifest (has digest) → Config (has ID) + Layers\"\n\n### Q2: How does layer caching work and why does COPY order matter?\n\n**Strong Answer:**\n\n\"Layer caching is Docker's optimization to avoid rebuilding unchanged layers.\n\nFor each instruction, Docker checks if it can reuse a cached layer:\n- `FROM`: Cache hit if base image digest matches\n- `RUN`: Cache hit if parent layer AND command string match\n- `COPY/ADD`: Cache hit if parent layer AND all source file contents match\n\nThe key insight: **cache invalidation cascades**. If layer N misses cache, all layers N+1, N+2... must rebuild.\n\nThis is why COPY order matters:\n\n```dockerfile\n# ❌ Bad - any code change invalidates npm install\nCOPY . .\nRUN npm install\n\n# ✅ Good - code changes don't invalidate npm install\nCOPY package*.json ./\nRUN npm install\nCOPY . .\n```\n\nIn the good version, `npm install` only reruns if `package.json` changes. The `COPY . .` for code changes doesn't affect it because it comes after.\n\nSame principle applies to any slow step: put dependencies before code, rarely-changing before frequently-changing.\"\n\n### Q3: What happens during `docker pull` at the network level?\n\n**Strong Answer:**\n\n\"Docker pull follows the OCI Distribution protocol:\n\n1. **Resolve tag to manifest**: GET to `/v2//manifests/`. Registry returns manifest JSON with config and layer digests.\n\n2. **Check for multi-arch**: If it's a manifest list, Docker selects the manifest matching local architecture.\n\n3. **Download config**: GET to `/v2//blobs/`. This is the image configuration JSON.\n\n4. **Download layers** (parallel):\n - For each layer, check local storage for matching digest\n - If missing: GET to `/v2//blobs/`\n - Response is gzipped tarball\n - Verify SHA256 matches expected digest\n - Extract to local storage\n\n5. **Assemble**: Create local image metadata linking config to layers.\n\nKey optimizations:\n- Layers download in parallel\n- Already-cached layers skip network entirely\n- Registries support range requests for resumable downloads\n- CDN acceleration for popular images\"\n\n### Q4: How would you debug an image that's larger than expected?\n\n**Strong Answer:**\n\n\"My debugging process:\n\n1. **Quick size check**:\n ```bash\n docker images myapp\n docker history myapp --no-trunc\n ```\n\n2. **Layer analysis with dive**:\n ```bash\n dive myapp\n ```\n This shows exactly which files are in each layer and flags wasted space.\n\n3. **Common issues I look for**:\n - Build artifacts in final image (node_modules dev deps, .git, test files)\n - Package manager cache not cleaned\n - Multiple RUN statements that could combine\n - Missing `.dockerignore`\n - Wrong base image (ubuntu vs alpine vs distroless)\n\n4. **Multi-stage check**:\n - Are we copying only needed artifacts?\n - Any unnecessary COPY commands?\n\n5. **Concrete fixes**:\n - Add `.dockerignore` for build context\n - Combine RUN commands with cleanup\n - Use smaller base image\n - Multi-stage build for compiled languages\n - Remove dev dependencies\n\nI'd also check if the team has image size targets in CI to prevent regression.\"\n\n### Q5: Explain content-addressable storage and why it matters.\n\n**Strong Answer:**\n\n\"Content-addressable storage means every blob is identified by the SHA256 hash of its contents. The hash IS the address.\n\nWhy this matters:\n\n1. **Deduplication**: If two images share a layer (same content = same hash), it's stored once. A host running 100 containers might only have 10 unique layers on disk.\n\n2. **Integrity**: When downloading `sha256:abc123`, you hash the received data. If it doesn't match, something went wrong (corruption, MITM). Immutable verification.\n\n3. **Immutability**: You can't modify a layer without changing its hash. This enables safe caching - if you have `sha256:abc123`, you know exactly what it contains, forever.\n\n4. **Efficient distribution**: Registries and clients can skip transferring layers they already have. `docker pull` is essentially 'sync these hashes'.\n\nThe tradeoff: tags become problematic. `nginx:latest` is content-addressed under the hood, but the tag can point to different digests over time. Production should use digest references: `nginx@sha256:abc...`\"\n\n---\n\n## 📝 Summary & Key Takeaways\n\n### Core Concepts\n\n| Concept | Key Point |\n|---------|-----------|\n| Layers | Filesystem diffs stacked with overlay FS |\n| Content-addressable | SHA256 hash = identity, enables deduplication |\n| Manifest | JSON listing config + layers, ties image together |\n| Multi-arch | Manifest list points to arch-specific manifests |\n| Registry protocol | Standard HTTP API for push/pull |\n\n### The Image Equation\n\n```\nImage = Manifest + Config + Layers\n\nManifest: \"Here's what this image contains\"\n → Config digest (how to run)\n → Layer digests (filesystem content)\n\nConfig: \"Here's how to run this image\"\n → CMD, ENV, EXPOSE, etc.\n → Build history\n\nLayers: \"Here's the filesystem\"\n → Ordered tarballs\n → Each is diff from previous\n → Stacked by overlay filesystem\n```\n\n### What You Can Do Now\n\n1. **Analyze images**: Use `docker history` and `dive` to understand composition\n2. **Debug size issues**: Identify which layers contribute most\n3. **Understand caching**: Know why builds are slow or fast\n4. **Use digests**: Reference immutable images in production\n\n---\n\n## 📋 Quick Reference\n\n### Image Inspection Commands\n\n```bash\n# Layer history\ndocker history IMAGE\n\n# Full metadata\ndocker inspect IMAGE\n\n# Manifest (including multi-arch)\ndocker manifest inspect IMAGE\n\n# On-disk location\ndocker inspect IMAGE --format '{{.GraphDriver.Data}}'\n\n# Export for analysis\ndocker save IMAGE -o image.tar\n```\n\n### Registry API Endpoints\n\n| Endpoint | Purpose |\n|----------|---------|\n| GET `/v2/_catalog` | List repositories |\n| GET `/v2//tags/list` | List tags |\n| GET `/v2//manifests/` | Get manifest |\n| GET `/v2//blobs/` | Get layer/config |\n| HEAD `/v2//blobs/` | Check if blob exists |\n\n### Disk Usage Commands\n\n```bash\n# Summary\ndocker system df\n\n# Detailed\ndocker system df -v\n\n# Clean unused\ndocker image prune # Dangling only\ndocker image prune -a # All unused\ndocker system prune -a # Everything unused\n```\n\n---\n\n## 📅 Review Schedule\n\n| Day | Task | Time |\n|-----|------|------|\n| Day 1 | Draw layer stacking diagram from memory | 10 min |\n| Day 3 | Do Exercise 1 (dissect an image) | 15 min |\n| Day 7 | Explain content-addressable storage to colleague | 5 min |\n| Day 14 | Do Exercise 4 (registry protocol) | 30 min |\n| Day 30 | Analyze a production image with dive | 20 min |\n\n---\n\n## 📚 Series Navigation\n\n| Previous | Current | Next |\n|----------|---------|------|\n| [Part 1: Container Internals](docker-part-1-container-internals) | **Part 2: Image Anatomy** | [Part 3: Build Process](docker-part-3-build-process) |\n\n**Docker Compendium Series:**\n- [Part 0: How to Use This Series](docker-part-0-how-to-use-this-series)\n- [Part 1: Container Internals](docker-part-1-container-internals)\n- **Part 2: Image Anatomy** ← You are here\n- [Part 3: Build Process Deep Dive](docker-part-3-build-process)\n- [Part 4: Networking Internals](docker-part-4-networking-internals)\n","featuredImage","isPublished","isFeatured","subscriptionType","BASIC","publishedAt","2026-06-10T07:19:25.386480Z","createdAt","2026-06-10T07:19:25.387155Z","updatedAt","2026-06-16T21:26:56.086200Z","readingTimeMinutes",19,"estimatedReadingTime","authorId",3,"authorUsername","tomasz_main","categoryName","Devops","readingTime","seriesId","seriesOrder",2,"author",{"_31":55,"_94":57,"_95":96,"_97":98,"_99":-5,"_100":-5,"_101":-5,"_102":-5,"_103":-5},"category",{"_31":90,"_72":59,"_35":91,"_75":92,"_93":-5},"tags",[78,79,80],"series",{"_31":55,"_72":73,"_35":74,"_75":76,"_77":23},"name","Docker Compendium: From Practitioner to Expert","docker-compendium","description","Master Docker internals, optimization patterns, and production best practices. For developers who use Docker daily but want deeper understanding of what's happening under the hood. Not a beginner series - we assume you can write Dockerfiles and use docker-compose. Learn what namespaces and cgroups really do, why your builds are slow, how to debug containers that crash, and how to run Docker in production like a pro.\n","isActive",{"_31":87,"_72":88,"_35":89},{"_31":84,"_72":85,"_35":86},{"_31":81,"_72":82,"_35":83},222,"Registry","registry",221,"Images","images",213,"Docker","docker",7,"devops","Category for Devops","colorHex","username","firstName","Tomasz","lastName","Jarosz","profilePicture","bio","website","githubUrl","linkedinUrl"]