Devops
Container Security Hardening
Containers provide isolation, but not security by default. Running as root, with all capabilities, writable filesystem - these defaults are convenient but dangerous. This article covers practical hardening: non-root users, read-only filesystems, capability dropping, and defense in depth.
š At a Glance
| Aspect | Details |
|---|---|
| Topic | Non-root users, capabilities, read-only fs, seccomp, AppArmor |
| Complexity | Advanced |
| Prerequisites | Part 1 (Container Internals), Part 7 (Base Images) |
| Key Insight | Security is layers - each control reduces attack surface |
| Time to Master | 3-4 hours |
šÆ What You'll Learn
- Non-root containers - why and how to run without root
- Linux capabilities - granular permissions instead of all-or-nothing
- Read-only filesystems - preventing runtime modifications
- Seccomp profiles - limiting system calls
- Defense in depth - combining multiple security controls
š„ Production Story: The Container Escape
A security audit found a critical vulnerability: an attacker who compromised the web application could escape the container and access the host.
The vulnerable setup:
YAML(6 lines)CodeLoading syntax highlighter...
The attack path:
BASH(4 lines)CodeLoading syntax highlighter...
The fix:
YAML(11 lines)CodeLoading syntax highlighter...
Lesson: Never use
privileged: true or mount host root. Every unnecessary permission is an attack vector.š§ Mental Model: Defense in Depth
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā CONTAINER SECURITY LAYERS ā ā ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā LAYER 1: IMAGE SECURITY ā ā ā ā¢ Minimal base image (distroless, alpine) ā ā ā ā¢ No secrets in layers ā ā ā ā¢ Regular vulnerability scanning ā ā ā ā¢ Signed/verified images ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā ā ā¼ ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā LAYER 2: USER PERMISSIONS ā ā ā ā¢ Non-root user ā ā ā ā¢ no-new-privileges ā ā ā ā¢ Minimal file permissions ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā ā ā¼ ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā LAYER 3: CAPABILITIES ā ā ā ā¢ Drop ALL capabilities ā ā ā ā¢ Add only what's needed ā ā ā ā¢ Never use --privileged ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā ā ā¼ ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā LAYER 4: FILESYSTEM ā ā ā ā¢ Read-only root filesystem ā ā ā ā¢ tmpfs for writable paths ā ā ā ā¢ No host mounts (or read-only) ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā ā ā¼ ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā LAYER 5: SYSCALL FILTERING ā ā ā ā¢ Seccomp profiles ā ā ā ā¢ AppArmor/SELinux ā ā ā ā¢ Block dangerous syscalls ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā ā ā¼ ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā LAYER 6: NETWORK ā ā ā ā¢ Isolated networks ā ā ā ā¢ No unnecessary port exposure ā ā ā ā¢ Network policies ā ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā ā ā Each layer reduces attack surface. Compromise of one layer ā ā doesn't mean full compromise. ā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
š¬ Deep Dive
Non-Root Containers
By default, containers run as root (UID 0). This is dangerous:
BASH(6 lines)CodeLoading syntax highlighter...
Create non-root user in Dockerfile:
DOCKERFILE(14 lines)CodeLoading syntax highlighter...
Run as non-root at runtime:
BASH(5 lines)CodeLoading syntax highlighter...
Docker Compose:
YAML(4 lines)CodeLoading syntax highlighter...
Verify non-root:
BASH(5 lines)CodeLoading syntax highlighter...
no-new-privileges
Prevents processes from gaining additional privileges:
BASH(13 lines)CodeLoading syntax highlighter...
Always enable:
YAML(4 lines)CodeLoading syntax highlighter...
Linux Capabilities
Capabilities split root power into granular permissions:
BASH(8 lines)CodeLoading syntax highlighter...
Common capabilities:
| Capability | Purpose |
|---|---|
| NET_BIND_SERVICE | Bind to ports < 1024 |
| NET_RAW | Use raw sockets (ping) |
| CHOWN | Change file ownership |
| SETUID/SETGID | Change user/group |
| SYS_ADMIN | Many admin operations (dangerous!) |
| SYS_PTRACE | Debug processes |
Drop all, add only needed:
BASH(5 lines)CodeLoading syntax highlighter...
YAML(7 lines)CodeLoading syntax highlighter...
Never use --privileged:
BASH(4 lines)CodeLoading syntax highlighter...
Read-Only Filesystem
Prevent runtime modifications to container filesystem:
BASHCodeLoading syntax highlighter...
Handle writable requirements:
BASH(9 lines)CodeLoading syntax highlighter...
YAML(7 lines)CodeLoading syntax highlighter...
Identify needed writable paths:
BASH(5 lines)CodeLoading syntax highlighter...
Seccomp Profiles
Seccomp filters system calls:
BASH(11 lines)CodeLoading syntax highlighter...
Custom seccomp profile:
JSON(12 lines)CodeLoading syntax highlighter...
Generate profile for your app:
BASH(6 lines)CodeLoading syntax highlighter...
AppArmor
AppArmor provides Mandatory Access Control:
BASH(11 lines)CodeLoading syntax highlighter...
Complete Hardened Container
DOCKERFILE(22 lines)CodeLoading syntax highlighter...
YAML(17 lines)CodeLoading syntax highlighter...
ā ļø Common Mistakes
Mistake 1: Running as Root
DOCKERFILE(11 lines)CodeLoading syntax highlighter...
Mistake 2: Using --privileged
YAML(10 lines)CodeLoading syntax highlighter...
Mistake 3: Mounting Sensitive Host Paths
YAML(12 lines)CodeLoading syntax highlighter...
š Debug This: The Permission Denied Mystery
A developer reports: "My app works with root but fails as non-root user!"
DOCKERFILE(7 lines)CodeLoading syntax highlighter...
BASH(2 lines)CodeLoading syntax highlighter...
Why does the non-root user fail?
ā
Solution:
The
/app directory and files are owned by root because COPY and npm install ran before USER instruction.The problem:
- COPY runs as root ā files owned by root
- npm install runs as root ā node_modules owned by root
- USER appuser switches user
- appuser can't write to root-owned directories
Fix 1: Set ownership explicitly:
DOCKERFILE(13 lines)CodeLoading syntax highlighter...
Fix 2: Create writable directories:
DOCKERFILE(11 lines)CodeLoading syntax highlighter...
Fix 3: Use tmpfs for temporary data:
YAML(5 lines)CodeLoading syntax highlighter...
Key lesson: Order matters. COPY ownership, directory permissions, and USER instruction must be coordinated.
š» Exercises
Exercise 1: Audit Container Security
ā Difficulty: Easy | ā±ļø Time: 15 minutes
BASH(19 lines)CodeLoading syntax highlighter...
Exercise 2: Create Non-Root Image
āā Difficulty: Medium | ā±ļø Time: 20 minutes
BASH(25 lines)CodeLoading syntax highlighter...
Exercise 3: Capability Analysis
āā Difficulty: Medium | ā±ļø Time: 20 minutes
BASH(16 lines)CodeLoading syntax highlighter...
Exercise 4: Read-Only Filesystem
āāā Difficulty: Hard | ā±ļø Time: 25 minutes
BASH(33 lines)CodeLoading syntax highlighter...
Exercise 5: Complete Hardened Setup
āāāā Difficulty: Expert | ā±ļø Time: 35 minutes
Create a fully hardened containerized application:
YAML(17 lines)CodeLoading syntax highlighter...
š¤ Senior-Level Interview Questions
Q1: Why should containers run as non-root?
Strong Answer:
"Running as root in containers is dangerous for several reasons:
1. Container escape amplification:
If an attacker exploits a container escape vulnerability, they're root on the host. Non-root limits the damage.
2. Defense in depth:
Even without escape, root can do more damage inside the container - modify binaries, read sensitive files, etc.
3. Shared kernel:
Container root is the same UID as host root. Kernel doesn't distinguish between 'root in container' and 'root on host'.
4. Best practices:
Most security frameworks (CIS Docker Benchmark, NIST) require non-root containers.
Implementation:
DOCKERFILE(4 lines)CodeLoading syntax highlighter...
Exceptions:
- Some software legitimately needs root (port 80, raw sockets)
- Solution: Drop privileges after initialization
- Or use capabilities instead of full root
I always start with non-root and only add specific capabilities if absolutely necessary."
Q2: Explain Linux capabilities and how they relate to container security.
Strong Answer:
"Linux capabilities break root's all-or-nothing power into 40+ granular permissions.
Traditional model:
- UID 0 = can do everything
- Other UIDs = restricted
Capabilities model:
- Each capability grants specific power
- Can have root UID without capabilities
- Can have capabilities without root UID
Key capabilities:
CAP_NET_BIND_SERVICE- Bind ports < 1024CAP_NET_RAW- Raw sockets (ping)CAP_SYS_ADMIN- Mount, namespace ops (dangerous!)CAP_CHOWN- Change file ownership
Docker default:
Containers get ~14 capabilities by default. Not minimal, but not full root either.
Best practice:
YAML(4 lines)CodeLoading syntax highlighter...
Why drop all:
- Reduces attack surface
- Most apps don't need any capabilities
- Forces explicit thought about requirements
The dangerous ones:
SYS_ADMIN- Basically root, avoidNET_ADMIN- Network configurationSYS_PTRACE- Process debugging
Never use
--privileged, which grants ALL capabilities plus more."Q3: How do you implement defense in depth for containers?
Strong Answer:
"Defense in depth means multiple security layers, so compromising one doesn't mean full compromise:
Layer 1 - Image security:
- Minimal base image (distroless, alpine)
- No secrets in image
- Regular vulnerability scanning
- Image signing
Layer 2 - Runtime security:
- Non-root user
- Read-only filesystem
- Drop all capabilities
- no-new-privileges
Layer 3 - System call filtering:
- Seccomp profiles (default or custom)
- Block dangerous syscalls
Layer 4 - Mandatory Access Control:
- AppArmor or SELinux profiles
- Restrict file access patterns
Layer 5 - Network security:
- Isolated networks
- Network policies
- No unnecessary ports
Layer 6 - Resource limits:
- Memory limits (prevent DoS)
- CPU limits
- PID limits
Example implementation:
YAML(15 lines)CodeLoading syntax highlighter...
Each layer stops different attack types. Even if one fails, others provide protection."
Q4: What is the difference between --privileged and specific capability grants?
Strong Answer:
"They're vastly different in security impact:
--privileged:
- Grants ALL capabilities (~40+)
- Gives access to all host devices
- Disables seccomp filtering
- Disables AppArmor
- Container is essentially root on host
- Use case: Almost never in production
Specific capabilities:
- Grant only what's needed
- Keep other restrictions in place
- Seccomp still active
- AppArmor still active
- Principle of least privilege
Example - needing raw sockets:
BASH(5 lines)CodeLoading syntax highlighter...
When people think they need privileged:
- Docker-in-Docker ā Use Docker socket mount or DinD image properly
- Device access ā Mount specific device
- Network tools ā Add NET_ADMIN or NET_RAW
- Debugging ā Add SYS_PTRACE temporarily
I've never seen a legitimate production use case for
--privileged. It's a security red flag in any deployment."Q5: How would you secure a container that needs to run as root?
Strong Answer:
"Sometimes root is truly needed (legacy apps, port 80, specific syscalls). Here's how to minimize risk:
1. Drop to non-root after initialization:
DOCKERFILE(3 lines)CodeLoading syntax highlighter...
BASH(7 lines)CodeLoading syntax highlighter...
2. Use capabilities instead:
YAML(5 lines)CodeLoading syntax highlighter...
3. Maximum restrictions:
YAML(12 lines)CodeLoading syntax highlighter...
4. User namespace remapping:
JSON(4 lines)CodeLoading syntax highlighter...
Root in container maps to unprivileged user on host.
5. Additional isolation:
- Custom seccomp profile
- AppArmor profile
- Read-only mounts
- Network isolation
The goal: If container is compromised, attacker gets minimal access even as root in container."
š Summary & Key Takeaways
Security Checklist
| Control | Implementation |
|---|---|
| Non-root user | USER appuser in Dockerfile |
| Drop capabilities | cap_drop: [ALL] |
| Read-only fs | read_only: true |
| No new privileges | no-new-privileges:true |
| Resource limits | memory, cpus limits |
| Minimal base | distroless, alpine |
Security Hierarchy
NEVER: --privileged, host mounts, docker.sock AVOID: Running as root, unnecessary capabilities ALWAYS: Non-root, dropped caps, read-only, limits
š Quick Reference
Docker Run Security Flags
BASH(15 lines)CodeLoading syntax highlighter...
Compose Security
YAML(10 lines)CodeLoading syntax highlighter...
š Review Schedule
| Day | Task | Time |
|---|---|---|
| Day 1 | Review security layers diagram | 10 min |
| Day 3 | Audit a running container's security | 15 min |
| Day 7 | Convert an image to non-root | 20 min |
| Day 14 | Implement read-only filesystem | 25 min |
| Day 30 | Full security hardening of production service | 45 min |
š Series Navigation
| Previous | Current | Next |
|---|---|---|
| Part 11: Logging & Observability | Part 12: Container Security | Part 13: Debugging Containers |