Devops

Networking Internals

Your containers can talk to each other by name. Port 8080 on localhost reaches your container. But how? This article reveals Docker's networking magic - from virtual ethernet pairs to iptables rules to DNS resolution. Understanding this transforms networking from mystery to manageable.

📋 At a Glance

AspectDetails
TopicBridge networks, iptables, DNS, port mapping, overlay networks
ComplexityAdvanced
PrerequisitesPart 1 (Container Internals), basic networking concepts
Key InsightDocker networking is just Linux networking with automation
Time to Master3-4 hours

🎯 What You'll Learn

  • Bridge networks - how containers on the same host communicate
  • Port mapping - how iptables routes traffic to containers
  • DNS resolution - how container names become IP addresses
  • Network drivers - bridge, host, none, overlay differences
  • Network debugging - tools and techniques for troubleshooting

🔥 Production Story: The Silent DNS Failure

A microservices application worked perfectly in development. In staging, services randomly couldn't reach each other. Logs showed connection timeouts with no pattern.

Investigation:
BASH(7 lines)
Code
Loading syntax highlighter...
Root cause: The staging environment ran Docker's embedded DNS on a host with a custom /etc/resolv.conf pointing to a corporate DNS server. Docker's DNS wasn't forwarding correctly, and with DNS caching, some containers resolved names while others failed.
The debugging journey:
BASH(11 lines)
Code
Loading syntax highlighter...
The fix:
YAML(8 lines)
Code
Loading syntax highlighter...
Lesson: Docker networking isn't magical. It uses standard Linux networking primitives that can fail in standard ways. Understanding the plumbing helps you debug.

🧠 Mental Model: Docker Network Stack

┌─────────────────────────────────────────────────────────────────────────┐
│                         HOST SYSTEM                                     │
│                                                                         │
│  ┌─────────────────────────────────────────────────────────────────────┐│
│  │                    HOST NETWORK NAMESPACE                           ││
│  │                                                                     ││
│  │  Physical:  eth0 ───────────────────────────────────► Internet      ││
│  │            192.168.1.100                                            ││
│  │                                                                     ││
│  │  Docker:   docker0 (bridge) ◄─────────────────────────────────────┐ ││
│  │            172.17.0.1                                             │ ││
│  │              │                                                    │ ││
│  │              ├─── veth123 ◄──────────────────────────────────┐    │ ││
│  │              │                                               │    │ ││
│  │              └─── veth456 ◄──────────────────────────────┐   │    │ ││
│  │                                                          │   │    │ ││
│  │  iptables:                                               │   │    │ ││
│  │    PREROUTING: -p tcp --dport 8080 → 172.17.0.2:80       │   │    │ ││
│  │    FORWARD: docker0 ↔ eth0 (masquerade)                  │   │    │ ││
│  │                                                          │   │    │ ││
│  └──────────────────────────────────────────────────────────┼───┼────┼─┘│
│                                                             │   │    │  │
│  ┌─────────────────────────────────────┐  ┌───────────────────────────┐ │
│  │      CONTAINER A NAMESPACE          │  │   CONTAINER B NAMESPACE   │ │
│  │                                     │  │                           │ │
│  │  eth0 ─────────────────────────────►│  │◄────────────────── eth0   │ │
│  │  172.17.0.2            (veth pair)  │  │  (veth pair)   172.17.0.3 │ │
│  │                                     │  │                           │ │
│  │  Sees:                              │  │  Sees:                    │ │
│  │   - Own eth0 (172.17.0.2)           │  │   - Own eth0 (172.17.0.3) │ │
│  │   - Gateway (172.17.0.1)            │  │   - Gateway (172.17.0.1)  │ │ 
│  │   - Can reach Container B           │  │   - Can reach Container A │ │
│  │                                     │  │                           │ │
│  └─────────────────────────────────────┘  └───────────────────────────┘ │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
Key insight: Each container has its own network namespace with a virtual ethernet interface. These connect to the host's docker0 bridge via veth pairs - like virtual network cables.

🔬 Deep Dive

Bridge Networks: The Default

When you install Docker, it creates a bridge network called bridge (device: docker0):
BASH(17 lines)
Code
Loading syntax highlighter...
How a container connects:
┌─────────────────────────────────────────────────────────────────┐
│                     docker run nginx                            │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 1. Create network namespace for container                       │
│    - Isolated network stack                                     │
│    - Own routing table, iptables, interfaces                    │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 2. Create veth pair (virtual ethernet cable)                    │
│    - One end: veth123 (in host namespace, attached to docker0)  │
│    - Other end: eth0 (in container namespace)                   │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 3. Assign IP address from bridge subnet                         │
│    - Container gets 172.17.0.X                                  │
│    - Gateway is 172.17.0.1 (the bridge)                         │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ 4. Configure DNS                                                │
│    - /etc/resolv.conf points to 127.0.0.11                      │
│    - Docker's embedded DNS server                               │
└─────────────────────────────────────────────────────────────────┘
See it happen:
BASH(15 lines)
Code
Loading syntax highlighter...

User-Defined Bridge Networks

The default bridge has limitations. User-defined bridges are better:

BASH(7 lines)
Code
Loading syntax highlighter...
Default bridge vs User-defined bridge:
FeatureDefault BridgeUser-Defined Bridge
DNS resolutionNo (IP only)Yes (by container name)
Automatic connectYesMust specify --network
Live connect/disconnectNoYes
Environment variable injectionYes (legacy)No
IsolationAll containersOnly containers on same network
BASH(16 lines)
Code
Loading syntax highlighter...

Port Mapping: How -p Works

When you run docker run -p 8080:80 nginx, Docker creates iptables rules:
BASH(9 lines)
Code
Loading syntax highlighter...
The full iptables flow:
┌─────────────────────────────────────────────────────────────────┐
│              INCOMING TRAFFIC TO HOST:8080                       │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ PREROUTING chain (nat table)                                    │
│                                                                 │
│ -A DOCKER -p tcp -m tcp --dport 8080                            │
│   -j DNAT --to-destination 172.17.0.2:80                        │
│                                                                 │
│ Destination rewritten: 172.17.0.2:80                            │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ FORWARD chain (filter table)                                    │
│                                                                 │
│ -A DOCKER -d 172.17.0.2 -p tcp --dport 80                       │
│   -j ACCEPT                                                     │
│                                                                 │
│ Traffic allowed to container                                    │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ docker0 bridge → veth → container eth0                          │
│                                                                 │
│ nginx receives request on port 80                               │
└─────────────────────────────────────────────────────────────────┘
Port mapping options:
BASH(14 lines)
Code
Loading syntax highlighter...

DNS Resolution: How Container Names Work

Docker runs an embedded DNS server at 127.0.0.11:

BASH(11 lines)
Code
Loading syntax highlighter...
DNS resolution flow:
┌─────────────────────────────────────────────────────────────────┐
│              Container queries: api-service                     │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ Container's /etc/resolv.conf                                    │
│ nameserver 127.0.0.11                                           │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│ Docker's embedded DNS server                                    │
│                                                                 │
│ 1. Check if name matches a container on same network            │
│    └─ YES: Return container's IP                                │
│    └─ NO: Forward to upstream DNS (from host)                   │
│                                                                 │
│ Features:                                                       │
│ - Container name → IP                                           │
│ - Service name → IP (for Swarm/Compose)                         │
│ - Network alias → IP                                            │
│ - External domains → forwarded                                  │
└─────────────────────────────────────────────────────────────────┘
Network aliases:
BASH(12 lines)
Code
Loading syntax highlighter...

Network Drivers

Docker supports multiple network drivers:

DriverUse CaseContainer Communication
bridgeDefault, single hostVia bridge, NAT for external
hostPerformance, no isolationShares host network stack
noneMaximum isolationNo networking
overlayMulti-host (Swarm)VXLAN tunnels
macvlanLegacy integrationDirect MAC on physical network
ipvlanLike macvlan, shared MACL2/L3 modes
Host network mode:
BASH(10 lines)
Code
Loading syntax highlighter...
None network mode:
BASH(6 lines)
Code
Loading syntax highlighter...
Overlay network (Swarm):
BASH(8 lines)
Code
Loading syntax highlighter...

Container-to-Container Communication

Same network (automatic):
BASH(7 lines)
Code
Loading syntax highlighter...
Different networks (need explicit connection):
BASH(11 lines)
Code
Loading syntax highlighter...
Cross-network communication pattern:
┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│  ┌────────────────────┐    ┌────────────────────┐               │
│  │   frontend (net)   │    │   backend (net)    │               │
│  │                    │    │                    │               │
│  │  ┌──────────────┐  │    │  ┌──────────────┐  │               │
│  │  │    nginx     │  │    │  │   postgres   │  │               │
│  │  │  (frontend)  │  │    │  │  (backend)   │  │               │
│  │  └──────────────┘  │    │  └──────────────┘  │               │
│  │                    │    │         ▲          │               │
│  │  ┌──────────────┐  │    │         │          │               │
│  │  │     api      ├──┼────┼─────────┘          │               │
│  │  │ (both nets!) │  │    │                    │               │
│  │  └──────────────┘  │    └────────────────────┘               │
│  │                    │                                         │
│  └────────────────────┘                                         │
│                                                                 │
│  nginx can reach api: YES (same network)                        │
│  api can reach postgres: YES (same network)                     │
│  nginx can reach postgres: NO (different networks)              │
└─────────────────────────────────────────────────────────────────┘

Network Debugging Tools

From inside container:
BASH(15 lines)
Code
Loading syntax highlighter...
From host:
BASH(22 lines)
Code
Loading syntax highlighter...
Debug container for network issues:
BASH(7 lines)
Code
Loading syntax highlighter...

⚠️ Common Mistakes

Mistake 1: Using Default Bridge for Inter-Container Communication

BASH(12 lines)
Code
Loading syntax highlighter...

Mistake 2: Exposing Ports Unnecessarily

BASH(9 lines)
Code
Loading syntax highlighter...

Mistake 3: Not Understanding Host Mode Implications

BASH(10 lines)
Code
Loading syntax highlighter...

🐛 Debug This: The Unreachable Service

A developer reports: "My web container can't reach the api container. They're on the same network!"

BASH(10 lines)
Code
Loading syntax highlighter...
Ping works but HTTP doesn't. Why?

✅ Solution:

The containers can reach each other (ping works), but the service isn't responding on port 8080.

Possible causes:
  1. API not listening on expected port:
BASH(7 lines)
Code
Loading syntax highlighter...
  1. API crashed or not started:
BASH(5 lines)
Code
Loading syntax highlighter...
  1. Firewall inside container:
BASH(2 lines)
Code
Loading syntax highlighter...
  1. API takes time to start:
BASH(6 lines)
Code
Loading syntax highlighter...
Debug steps:
BASH(16 lines)
Code
Loading syntax highlighter...
Most common cause: Application binding to 127.0.0.1 (localhost) instead of 0.0.0.0 (all interfaces).

💻 Exercises

Exercise 1: Explore Default Bridge

⭐ Difficulty: Easy | ⏱️ Time: 15 minutes

BASH(23 lines)
Code
Loading syntax highlighter...

Exercise 2: Build Isolated Network Topology

⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes

BASH(36 lines)
Code
Loading syntax highlighter...

Exercise 3: Analyze Port Mapping

⭐⭐ Difficulty: Medium | ⏱️ Time: 20 minutes

BASH(21 lines)
Code
Loading syntax highlighter...

Exercise 4: DNS Deep Dive

⭐⭐⭐ Difficulty: Hard | ⏱️ Time: 25 minutes

BASH(28 lines)
Code
Loading syntax highlighter...

Exercise 5: Network Namespace Surgery

⭐⭐⭐⭐ Difficulty: Expert | ⏱️ Time: 30 minutes

BASH(37 lines)
Code
Loading syntax highlighter...

🎤 Senior-Level Interview Questions

Q1: Explain how Docker networking works when you run docker run -p 8080:80 nginx.

Strong Answer:

"When you run this command, Docker does several things:

  1. Creates a network namespace for the container with its own network stack.
  2. Creates a veth pair - a virtual network cable:
    • One end (eth0) goes inside the container's namespace
    • Other end (veth*) attaches to docker0 bridge on host
  3. Assigns IP from docker0's subnet (typically 172.17.0.x)
  4. Sets up iptables rules for port mapping:
    • NAT PREROUTING/DOCKER chain: DNAT rule rewrites destination from host:8080 to container-ip:80
    • FILTER FORWARD/DOCKER chain: Allows traffic to container
  5. Configures container's routing - gateway points to docker0 (172.17.0.1)

Traffic flow: External → host:8080 → iptables DNAT → docker0 bridge → veth pair → container:80

For outbound traffic, iptables MASQUERADE rewrites source IP to host's IP, enabling internet access."

Q2: Why does container DNS not work on the default bridge but works on user-defined bridges?

Strong Answer:

"This is a deliberate Docker design decision:

The default bridge (docker0) predates user-defined networks and uses legacy features like --link for container discovery. For backward compatibility, DNS-based service discovery was not enabled.

User-defined bridges run Docker's embedded DNS server (127.0.0.11) and configure containers to use it. This DNS server maintains a mapping of container names and network aliases to IP addresses.

The embedded DNS:

  • Intercepts DNS queries via iptables rules
  • Resolves container names within the same network
  • Forwards external queries to host's DNS servers

Additional benefits of user-defined bridges:

  • Network isolation between networks
  • Containers can be connected/disconnected at runtime
  • Support for network aliases
  • Better security - no automatic connectivity between networks

I always recommend user-defined networks for any multi-container application."

Q3: How would you debug network connectivity issues between containers?

Strong Answer:

"My systematic debugging approach:

  1. Verify containers are on the same network:
    BASH
    Code
    Loading syntax highlighter...
  2. Check if containers are running:
    BASH
    Code
    Loading syntax highlighter...
  3. Test basic connectivity (ping):
    BASH
    Code
    Loading syntax highlighter...
  4. If ping works but app doesn't, check the service:
    BASH(2 lines)
    Code
    Loading syntax highlighter...
  5. Check DNS resolution:
    BASH
    Code
    Loading syntax highlighter...
  6. Check from inside target:
    BASH
    Code
    Loading syntax highlighter...
  7. Use a debug container:
    BASH(2 lines)
    Code
    Loading syntax highlighter...

Common issues:

  • App binding to 127.0.0.1 instead of 0.0.0.0
  • Wrong network
  • Service not started
  • Firewall rules"

Q4: What's the difference between bridge, host, and overlay network modes?

Strong Answer:

"Each serves different use cases:

Bridge (default):
  • Creates isolated network namespace per container
  • Containers communicate via virtual bridge
  • Port mapping required for external access
  • Most common, good isolation
  • Use for: Single-host applications, typical web apps
Host:
  • Container shares host's network namespace
  • No network isolation
  • No port mapping needed (direct access to host ports)
  • Better performance (no NAT overhead)
  • Use for: Network monitoring tools, performance-critical apps
Overlay:
  • Spans multiple Docker hosts
  • Uses VXLAN encapsulation
  • Requires Swarm or external key-value store
  • Built-in encryption available
  • Use for: Multi-host clustering, Docker Swarm services

There's also:

  • none: No networking, maximum isolation
  • macvlan: Container gets MAC on physical network (legacy integration)

Selection criteria:

  • Single host, need isolation → bridge
  • Multi-host → overlay
  • Maximum performance → host
  • Legacy network integration → macvlan"

Q5: How does Docker handle DNS resolution for containers?

Strong Answer:

"Docker runs an embedded DNS server on 127.0.0.11 for user-defined networks.

Resolution flow:
  1. Container queries 127.0.0.11 (configured in /etc/resolv.conf)
  2. Docker's DNS checks internal registry:
    • Container names on same network
    • Network aliases
    • Service names (in Swarm)
  3. If no internal match, forwards to upstream DNS (from host's /etc/resolv.conf)
Implementation details:
  • DNS server runs in dockerd process
  • iptables rules redirect port 53 traffic
  • Results are cached (short TTL for container IPs)
  • Multiple names can resolve to same IP (aliases)
Customization options:
BASH(8 lines)
Code
Loading syntax highlighter...
Gotchas:
  • Only works on user-defined networks (not default bridge)
  • Container names must be DNS-valid
  • DNS resolution can fail if embedded DNS is slow (add timeout options)
  • Swarm services get load-balanced DNS (VIP mode by default)"

📝 Summary & Key Takeaways

Core Concepts

ConceptKey Point
Bridge networkVirtual switch connecting containers, uses veth pairs
Port mappingiptables DNAT rules route traffic to containers
DNS resolutionEmbedded DNS on 127.0.0.11 for user-defined networks
Network namespaceEach container has isolated network stack
Network driversbridge, host, none, overlay for different use cases

The Network Equation

Container Network = Namespace + veth pair + Bridge + iptables + DNS

Namespace: Isolated network stack
Veth pair: Virtual cable to bridge
Bridge: Switch connecting containers
Iptables: Port mapping and NAT
DNS: Name resolution (user-defined networks)

What You Can Do Now

  1. Use user-defined networks: Always prefer over default bridge
  2. Debug connectivity: Follow systematic approach
  3. Understand port mapping: Know what iptables rules do
  4. Isolate properly: Use multiple networks for security

📋 Quick Reference

Network Commands

BASH(17 lines)
Code
Loading syntax highlighter...

Debugging Commands

BASH(15 lines)
Code
Loading syntax highlighter...

Port Mapping Options

BASH(14 lines)
Code
Loading syntax highlighter...

📅 Review Schedule

DayTaskTime
Day 1Draw network diagram (bridge, veth, container)10 min
Day 3Do Exercise 1 (explore default bridge)15 min
Day 7Build isolated topology from Exercise 220 min
Day 14Debug a network issue in real project20 min
Day 30Explain Docker networking to colleague10 min

📚 Series Navigation

PreviousCurrentNext
Part 3: Build ProcessPart 4: Networking InternalsPart 5: Dockerfile Optimization
Docker Compendium Series: