Docker in Production: BuildKit, Compose v2, Multi-stage and Security

Docker in development vs Docker in production

Most teams use Docker in development without difficulty. A docker compose up, a few mounted volumes, and everything works. But moving to production reveals very different challenges: build time, image size, security, resource management, monitoring and reliability.

In development, you tolerate 2 GB images, 10-minute builds and containers running as root. In production, every megabyte counts, every second of build impacts the deployment cycle, and every root container is a potential vulnerability.

This guide covers the techniques and tools essential to running Docker professionally: BuildKit for fast and secure builds, multi-stage builds for minimal images, Compose v2 for orchestration, and the best practices for security, monitoring and networking.

BuildKit: the next-generation build engine

BuildKit has been the default build engine since Docker 23.0 (January 2023). It replaces the legacy builder and brings major improvements in terms of performance and security. If you are using a recent version of Docker, BuildKit is already active.

Enabling and configuring

Check that BuildKit is properly enabled on your system:

# Check the version and the active builder
docker buildx version
docker buildx ls

# Force BuildKit for older versions
export DOCKER_BUILDKIT=1

# Or in /etc/docker/daemon.json
{
  "features": {
    "buildkit": true
  }
}

Create a dedicated builder with advanced options for production:

# Create a builder with a larger cache
docker buildx create --name production-builder \
  --driver docker-container \
  --config /etc/buildkitd.toml \
  --use

# buildkitd.toml configuration
[worker.oci]
  max-parallelism = 4
  gc = true
  gckeepstorage = 10000  # 10 GB of cache
  [[worker.oci.gcpolicy]]
    keepBytes = 5000000000  # 5 GB minimum
    keepDuration = 604800   # 7 days

Cache mounts: speeding up builds

Cache mounts are the most impactful BuildKit feature. They allow package manager caches to persist between builds, eliminating repetitive downloads.

# syntax=docker/dockerfile:1

FROM python:3.12-slim AS builder

WORKDIR /app
COPY requirements.txt .

# Cache mount for pip: downloaded packages are reused
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-compile -r requirements.txt

# Equivalent for Node.js
FROM node:20-alpine AS node-builder
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --production

# Equivalent for apt
FROM debian:bookworm-slim
RUN --mount=type=cache,target=/var/cache/apt \
    --mount=type=cache,target=/var/lib/apt/lists \
    apt-get update && apt-get install -y --no-install-recommends \
    curl ca-certificates

Secret mounts: injecting credentials without exposing them

The classic problem: your build needs a token to access a private registry, a Git repo, or an API. With the old builder, the secret ended up in an image layer. BuildKit solves this problem with secret mounts.

# syntax=docker/dockerfile:1

FROM python:3.12-slim

# The secret is mounted read-only, never stored in a layer
RUN --mount=type=secret,id=pip_conf,target=/etc/pip.conf \
    pip install -r requirements.txt

# Access to a private repo via SSH
RUN --mount=type=ssh \
    git clone [email protected]:private/repo.git

# Pass the secret to the build
docker buildx build \
  --secret id=pip_conf,src=$HOME/.pip/pip.conf \
  --ssh default=$SSH_AUTH_SOCK \
  -t myapp:latest .

Parallel builds and distributed cache

BuildKit automatically parallelizes the independent stages of a Dockerfile. Combined with a distributed cache via a registry, builds become fast even on ephemeral CI runners.

# Build with distributed cache via a registry
docker buildx build \
  --cache-from type=registry,ref=registry.example.com/myapp:cache \
  --cache-to type=registry,ref=registry.example.com/myapp:cache,mode=max \
  -t myapp:latest \
  --push .

# Local cache for frequent builds
docker buildx build \
  --cache-from type=local,src=/tmp/buildcache \
  --cache-to type=local,dest=/tmp/buildcache,mode=max \
  -t myapp:latest .

The max mode exports all the intermediate layers, not just those of the final image. This maximizes cache hits for the build stages.