CI/CD Pipeline Patterns

📋 At a Glance

Aspect	Details
Difficulty	🟠 Advanced
Prerequisites	Part 3 (Build Process), Part 6 (Multi-Stage)
CI Platforms	GitHub Actions, GitLab CI, Jenkins
Time Investment	28 minutes read + 90 minutes practice
Payoff	10x faster builds, secure deployments, automated releases

🎯 What You'll Learn

After this article, you'll be able to:

Optimize CI/CD build speed with intelligent caching strategies
Build multi-architecture images for ARM and x86
Implement security scanning in your pipeline
Tag and version images properly for production
Deploy with confidence using rolling updates and rollbacks

🔥 Production Story: The 45-Minute Build

The Setup: A team's CI pipeline took 45 minutes per build. Developers avoided pushing because feedback was too slow. PRs piled up.

The Pipeline:

YAML(3 lines)
Code
Loading syntax highlighter...

The Investigation:

Step 1/15 : FROM node:20
 ---> Pulling image (30 seconds)
Step 5/15 : RUN npm ci
 ---> Installing all packages (15 minutes)
Step 10/15 : RUN npm run build
 ---> Building (10 minutes)

Root Causes:

No Docker layer caching between builds
No dependency caching (npm ci every time)
Single-stage build (dev dependencies in production)
Building same image for test and deploy

The Fix:

YAML(12 lines)
Code
Loading syntax highlighter...

Results:

First build: 15 minutes (vs 45)
Cached builds: 3 minutes
90% reduction in CI time

🧠 Mental Model: CI/CD Pipeline Stages

┌─────────────────────────────────────────────────────────────────┐
│                    CI/CD PIPELINE FLOW                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   ┌──────────────────────────────────────────────────────┐      │
│   │                      BUILD                           │      │
│   │  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐  │      │
│   │  │  Lint   │→ │  Test   │→ │  Build  │→ │  Scan   │  │      │
│   │  │ (fast)  │  │  unit   │  │  image  │  │security │  │      │
│   │  └─────────┘  └─────────┘  └─────────┘  └─────────┘  │      │
│   └──────────────────────────────────────────────────────┘      │
│                           ↓                                     │
│   ┌──────────────────────────────────────────────────────┐      │
│   │                      TEST                            │      │
│   │  ┌─────────┐  ┌─────────┐  ┌─────────┐               │      │
│   │  │  E2E    │→ │  Int    │→ │  Perf   │               │      │
│   │  │  tests  │  │  tests  │  │  tests  │               │      │
│   │  └─────────┘  └─────────┘  └─────────┘               │      │
│   └──────────────────────────────────────────────────────┘      │
│                           ↓                                     │
│   ┌──────────────────────────────────────────────────────┐      │
│   │                     RELEASE                          │      │
│   │  ┌─────────┐  ┌─────────┐  ┌─────────┐               │      │
│   │  │  Tag    │→ │  Push   │→ │ Deploy  │               │      │
│   │  │ version │  │  to reg │  │  to env │               │      │
│   │  └─────────┘  └─────────┘  └─────────┘               │      │
│   └──────────────────────────────────────────────────────┘      │
│                                                                 │
│   CACHING STRATEGY                                              │
│   ┌──────────────────────────────────────────────────────┐      │
│   │  Layer Cache ─→ Dependency Cache ─→ Build Cache      │      │
│   │                                                      │      │
│   │  Registry      npm/maven         BuildKit            │      │
│   │  cache-from    cache action      --mount=type=cache  │      │
│   └──────────────────────────────────────────────────────┘      │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

🔬 Deep Dive

1. GitHub Actions - Complete Pipeline

YAML(188 lines)
Code
Loading syntax highlighter...

2. Caching Strategies

Layer Caching with BuildKit:

YAML(22 lines)
Code
Loading syntax highlighter...

Dependency Caching in Dockerfile:

DOCKERFILE(35 lines)
Code
Loading syntax highlighter...

Maven/Gradle Caching:

DOCKERFILE(7 lines)
Code
Loading syntax highlighter...

3. Multi-Architecture Builds

Building for ARM and x86:

YAML(9 lines)
Code
Loading syntax highlighter...

Dockerfile Considerations for Multi-Arch:

DOCKERFILE(20 lines)
Code
Loading syntax highlighter...

GitLab CI Multi-Arch:

YAML(14 lines)
Code
Loading syntax highlighter...

4. Image Tagging Strategies

Semantic Versioning:

YAML(21 lines)
Code
Loading syntax highlighter...

Generated Tags Example:

Event	Tags Generated
Push to `develop`	`develop`, `sha-abc1234`
PR #42	`pr-42`
Tag `v1.2.3`	`1.2.3`, `1.2`, `1`, `latest`, `sha-abc1234`

Manual Tagging Script:

BASH(21 lines)
Code
Loading syntax highlighter...

5. Security Scanning

Trivy Scanner in Pipeline:

YAML(24 lines)
Code
Loading syntax highlighter...

Snyk Integration:

YAML(8 lines)
Code
Loading syntax highlighter...

Scan During Build:

DOCKERFILE(14 lines)
Code
Loading syntax highlighter...

6. GitLab CI Complete Pipeline

YAML(129 lines)
Code
Loading syntax highlighter...

7. Jenkins Pipeline

GROOVY(79 lines)
Code
Loading syntax highlighter...

8. Build Optimization Checklist

YAML(71 lines)
Code
Loading syntax highlighter...

⚠️ Common Mistakes

Mistake 1: No Caching Strategy

YAML(8 lines)
Code
Loading syntax highlighter...

Mistake 2: Building Unnecessarily

YAML(11 lines)
Code
Loading syntax highlighter...

Mistake 3: Using Mutable Tags

YAML(6 lines)
Code
Loading syntax highlighter...

Mistake 4: No Security Scanning

YAML(9 lines)
Code
Loading syntax highlighter...

Mistake 5: Secrets in Build Args

YAML(5 lines)
Code
Loading syntax highlighter...

DOCKERFILE(3 lines)
Code
Loading syntax highlighter...

🐛 Debug This

CI builds are slow, even with caching enabled:

YAML(6 lines)
Code
Loading syntax highlighter...

DOCKERFILE(6 lines)
Code
Loading syntax highlighter...

Build always runs npm ci even when only source files changed. Why?

Click to reveal analysis

Problem: COPY . . includes everything, including source files. When any file changes, the cache invalidates, forcing npm ci to run again.

Fix: Order layers by change frequency:

DOCKERFILE(16 lines)
Code
Loading syntax highlighter...

Additional optimization: Add .dockerignore:

node_modules
.git
*.md
.env*

Now npm ci only runs when package.json or package-lock.json changes.

💻 Exercises

Exercise 1: Basic CI Pipeline

Create a GitHub Actions workflow that:

Runs lint and tests
Builds Docker image
Pushes to GitHub Container Registry
Uses caching

Exercise 2: Multi-Arch Build

Modify your pipeline to:

Build for both AMD64 and ARM64
Use QEMU for cross-platform builds
Push multi-arch manifest

Exercise 3: Security Scanning

Add security scanning that:

Scans for HIGH and CRITICAL vulnerabilities
Fails the build on findings
Uploads results to GitHub Security tab

Exercise 4: Tagging Strategy

Implement automatic tagging:

Branch builds: branch-name, sha-xxx
Tag builds: 1.2.3, 1.2, 1, latest
PR builds: pr-123

Exercise 5: Complete Pipeline

Create a full pipeline with:

Lint, unit tests, integration tests
Docker build with caching
Security scan
Deploy to staging (on develop)
Deploy to production (on tag, manual approval)

🎤 Interview Questions

Q1: How do you optimize Docker build times in CI/CD?

Answer: Multiple strategies at different levels:

BuildKit Layer Caching:

YAML(4 lines)
Code
Loading syntax highlighter...

Dockerfile Layer Ordering:

DOCKERFILE(5 lines)
Code
Loading syntax highlighter...

BuildKit Cache Mounts:

DOCKERFILE
Code
Loading syntax highlighter...

Parallelization:

Multi-stage builds run in parallel when possible
Split independent tests into parallel jobs

Build Context Optimization:

Use .dockerignore to exclude unnecessary files
Only copy what's needed

Q2: How do you handle secrets in CI/CD Docker builds?

Answer: Never expose secrets in the image or build args:

BuildKit Secrets (recommended):

YAML(9 lines)
Code
Loading syntax highlighter...

Build-Time Only:

DOCKERFILE(8 lines)
Code
Loading syntax highlighter...

Runtime Injection:

YAML(5 lines)
Code
Loading syntax highlighter...

Q3: Explain how you would implement a multi-architecture build pipeline.

Answer:

Setup QEMU and BuildX:

YAML(2 lines)
Code
Loading syntax highlighter...

Build with platforms:

YAML(4 lines)
Code
Loading syntax highlighter...

Dockerfile Considerations:

DOCKERFILE(10 lines)
Code
Loading syntax highlighter...

Result: Single tag (myapp:1.0) contains manifest with both architectures. Docker automatically pulls the right one for the host.

Q4: How do you integrate security scanning into a Docker CI/CD pipeline?

Answer: Multiple layers of security:

Image Scanning (vulnerabilities in dependencies):

YAML(5 lines)
Code
Loading syntax highlighter...

Dockerfile Linting (configuration issues):

YAML(3 lines)
Code
Loading syntax highlighter...

Secret Scanning (leaked credentials):

YAML(3 lines)
Code
Loading syntax highlighter...

SBOM Generation (software bill of materials):

YAML(3 lines)
Code
Loading syntax highlighter...

Policy Enforcement:

Fail build on CRITICAL vulnerabilities
Allow HIGH with exceptions
Block images without security scan

Q5: What's your strategy for image tagging in a CI/CD pipeline?

Answer: Tagging strategy depends on use case:

Branch Builds (development):

myapp:develop
myapp:feature-xyz
myapp:sha-abc1234

Release Builds (production):

myapp:1.2.3        # Exact version
myapp:1.2          # Latest patch
myapp:1            # Latest minor
myapp:latest       # Current stable

Implementation:

YAML(7 lines)
Code
Loading syntax highlighter...

Best Practices:

Never rely solely on latest for deployments
Use immutable tags (SHA) for traceability
Semantic versions for human readability
Keep audit trail of what's deployed where

📝 Summary & Key Takeaways

Caching Strategy

Use BuildKit cache (type=gha or type=registry)
Order Dockerfile layers by change frequency
Use cache mounts for package managers
Optimize build context with .dockerignore

Tagging Strategy

Branch: branch-name, sha-xxx
Release: 1.2.3, 1.2, 1, latest
Always use SHA for deployment tracking

Security

Scan images before pushing
Use BuildKit secrets for sensitive data
Generate SBOMs for compliance

Pipeline Stages

Lint → Test → Build → Scan → Deploy

📋 Quick Reference

YAML(19 lines)
Code
Loading syntax highlighter...

📅 Review Schedule

Day 1: Set up basic build and push pipeline
Day 3: Add caching strategies
Day 7: Implement security scanning
Day 14: Build multi-arch images
Day 30: Optimize and add deployment stages

Previous: Part 16 - Development vs Production Compose
Next: Part 18 - Production Deployment Patterns
Index: Docker Compendium Series