GitLab CI/CD: Continuous Integration Pipeline

Prerequisites

• GitLab: A GitLab.com instance (SaaS) or self-hosted GitLab (version 14.0 minimum recommended)
• Git repository: A GitLab project with an initialized Git repository
• Runner: Access to a GitLab Runner (shared runners available on GitLab.com)
• Knowledge: Basics of Git, YAML and the Linux command line
• Docker (optional): For jobs based on Docker containers

GitLab CI/CD Architecture

Before configuring your first pipeline, it is essential to understand the fundamental components of the GitLab CI/CD architecture and how they interact.

The main components

• Pipeline: The entire CI/CD process triggered by an event (commit, merge request, schedule). A pipeline contains one or more stages executed sequentially.
• Stage: A logical step of the pipeline (build, test, deploy). All jobs within the same stage run in parallel.
• Job: The elementary unit of work. Each job contains a script and runs on a runner. A job belongs to exactly one stage.
• Runner: The execution agent that handles jobs. It can run on a physical machine, a VM or a Docker container.

Runner types

GitLab offers three levels of runners depending on the desired scope:

• Shared runners: Available to all projects on the instance. Ideal for GitLab.com.
• Group runners: Dedicated to a GitLab group and its subprojects.
• Project runners: Assigned to a single project. Recommended for specific needs (GPU, internal network access).

Installing a GitLab Runner

If the shared runners are not enough, you can install your own runner to gain more control and performance.

Installation on Linux

# Add the official GitLab Runner repository
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# Install the runner
sudo apt install -y gitlab-runner

# Verify the installation
gitlab-runner --version

Registering the runner

After installation, register the runner with your GitLab instance. Get the registration token from Settings > CI/CD > Runners of your project:

# Interactive registration
sudo gitlab-runner register

# Or in a single command (non-interactive)
sudo gitlab-runner register
  --non-interactive
  --url "https://gitlab.example.com/"
  --registration-token "YOUR_TOKEN"
  --executor "docker"
  --docker-image "alpine:latest"
  --description "runner-docker-01"
  --tag-list "docker,linux,ci"
  --run-untagged="true"

Tags and runner selection

Tags let you direct jobs to specific runners. Use them to target runners with particular capabilities:

# Job that requires a runner with a GPU
train_model:
  stage: build
  tags:
    - gpu
    - linux
  script:
    - python train.py

The .gitlab-ci.yml file

The .gitlab-ci.yml file is the heart of your CI/CD configuration. Placed at the root of your repository, it defines your entire pipeline.

Basic structure

# Definition of stages (execution order)
stages:
  - build
  - test
  - deploy

# Global variables
variables:
  APP_NAME: "my-application"
  NODE_VERSION: "20"

# Default configuration for all jobs
default:
  image: node:20-alpine
  before_script:
    - echo "Start of job $CI_JOB_NAME"

# Build job
build_app:
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

# Test job
test_unit:
  stage: test
  script:
    - npm ci
    - npm run test

# Deployment job
deploy_prod:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - ./deploy.sh
  only:
    - main

Essential keywords

• image: The Docker image used to run the job
• stage: The stage the job belongs to
• script: The commands to run (mandatory)
• before_script / after_script: Commands run before/after the main script
• artifacts: Files to keep between stages
• only / except: Execution conditions (deprecated, prefer rules)
• tags: Runner selection by tags

Multi-stage pipelines

A typical production pipeline has several stages that run sequentially, each grouping jobs that run in parallel.

stages:
  - install
  - build
  - test
  - quality
  - staging
  - production

# --- STAGE: install ---
install_deps:
  stage: install
  image: node:20-alpine
  script:
    - npm ci --cache .npm
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - .npm/
      - node_modules/

# --- STAGE: build ---
build_frontend:
  stage: build
  image: node:20-alpine
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 2 hours

build_backend:
  stage: build
  image: golang:1.22-alpine
  script:
    - go build -o bin/api ./cmd/api
  artifacts:
    paths:
      - bin/
    expire_in: 2 hours

# --- STAGE: test ---
test_unit:
  stage: test
  script:
    - npm run test:unit -- --coverage
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'
  artifacts:
    reports:
      junit: reports/junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

test_integration:
  stage: test
  services:
    - postgres:16-alpine
  variables:
    POSTGRES_DB: test_db
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: secret
  script:
    - npm run test:integration

# --- STAGE: quality ---
lint_code:
  stage: quality
  script:
    - npm run lint
  allow_failure: true

# --- STAGE: staging ---
deploy_staging:
  stage: staging
  script:
    - ./scripts/deploy.sh staging
  environment:
    name: staging
    url: https://staging.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == "develop"

# --- STAGE: production ---
deploy_production:
  stage: production
  script:
    - ./scripts/deploy.sh production
  environment:
    name: production
    url: https://example.com
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  when: manual

Variables and Secrets

GitLab CI/CD provides a complete variable management system to parameterize your pipelines and protect your secrets.

Predefined variables

GitLab automatically injects many variables into each job:

debug_info:
  stage: build
  script:
    - echo "Branch    : $CI_COMMIT_BRANCH"
    - echo "Commit    : $CI_COMMIT_SHA"
    - echo "Pipeline  : $CI_PIPELINE_ID"
    - echo "Project   : $CI_PROJECT_NAME"
    - echo "Author    : $CI_COMMIT_AUTHOR"
    - echo "MR        : $CI_MERGE_REQUEST_IID"
    - echo "Runner    : $CI_RUNNER_DESCRIPTION"

CI/CD variables (Settings)

Define your secrets in Settings > CI/CD > Variables. Two essential protection options:

• Protected: The variable is injected only into jobs running on protected branches or tags (main, release/*).
• Masked: The value is replaced by [MASKED] in the pipeline logs. The value must be at least 8 characters long.

deploy_app:
  stage: deploy
  script:
    # $DEPLOY_TOKEN is defined in Settings > CI/CD > Variables
    # with the Protected + Masked options enabled
    - echo "$DEPLOY_TOKEN" | docker login registry.example.com -u deploy --password-stdin
    - docker push registry.example.com/$CI_PROJECT_PATH:$CI_COMMIT_SHA
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Cache and Artifacts

Cache and artifacts are two distinct mechanisms for optimizing the speed of your pipelines and sharing files between jobs.

Cache: speeding up builds

The cache persists between pipeline runs. Use it for dependencies that rarely change:

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.pip-cache"

build_python:
  stage: build
  image: python:3.12-slim
  cache:
    key:
      files:
        - requirements.txt
    paths:
      - .pip-cache/
      - venv/
  script:
    - python -m venv venv
    - source venv/bin/activate
    - pip install -r requirements.txt
    - python setup.py build

Artifacts: sharing between stages

Artifacts are files produced by a job and passed on to subsequent stages:

build_app:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
      - package.json
    exclude:
      - dist/**/*.map
    expire_in: 1 week
    when: on_success

test_e2e:
  stage: test
  # The artifacts from build_app are automatically available
  script:
    - ls dist/   # The dist/ directory is present
    - npm run test:e2e
  artifacts:
    when: on_failure
    paths:
      - screenshots/
    expire_in: 3 days

Docker in CI/CD

Building Docker images in a CI/CD pipeline is a common need. Two main approaches exist, each with its advantages and trade-offs.

Docker-in-Docker (DinD)

DinD launches a full Docker daemon inside the runner. Simple, but it requires privileged mode:

build_image_dind:
  stage: build
  image: docker:26
  services:
    - docker:26-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  before_script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY -u $CI_REGISTRY_USER --password-stdin

Kaniko (recommended)

Kaniko builds Docker images without a daemon and without elevated privileges, which makes it the safest option:

build_image_kaniko:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.0-debug
    entrypoint: [""]
  script:
    - mkdir -p /kaniko/.docker
    - |
      echo "{"auths":{"$CI_REGISTRY":{"auth":"$(echo -n $CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD | base64)"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
      --destination $CI_REGISTRY_IMAGE:latest
      --cache=true

Automated testing

Continuous integration truly shines with an automated test suite run on every commit. GitLab CI/CD offers advanced features for test reports.

Unit tests with coverage

test_unit:
  stage: test
  image: node:20-alpine
  script:
    - npm ci
    - npm run test:unit -- --coverage --reporters=default --reporters=jest-junit
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
    paths:
      - coverage/
    when: always

Python tests with pytest

test_python:
  stage: test
  image: python:3.12-slim
  before_script:
    - pip install -r requirements.txt
    - pip install pytest pytest-cov pytest-html
  script:
    - pytest tests/
      --cov=app
      --cov-report=xml:coverage.xml
      --cov-report=html:htmlcov/
      --junitxml=report.xml
  coverage: '/TOTAL.*\s+(\d+%)$/'
  artifacts:
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
    paths:
      - htmlcov/
    when: always

Reports in Merge Requests

By configuring JUnit and Cobertura reports, GitLab automatically displays the test results and coverage directly in the merge requests. The coverage badge is configured in Settings > CI/CD > General pipelines > Test coverage parsing.

Deployment

GitLab CI/CD includes an environments system that provides full traceability of deployments along with advanced features such as review apps and rollback.

Environments and Review Apps

# Review app: ephemeral environment per merge request
review_app:
  stage: staging
  script:
    - kubectl apply -f k8s/review/
    - kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: https://$CI_COMMIT_REF_SLUG.review.example.com
    on_stop: stop_review
    auto_stop_in: 1 week
  rules:
    - if: $CI_MERGE_REQUEST_IID

stop_review:
  stage: staging
  script:
    - kubectl delete namespace review-$CI_COMMIT_REF_SLUG
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: manual

Production deployment with manual approval

deploy_production:
  stage: production
  script:
    - echo "Deploying to production..."
    - ansible-playbook -i inventory/prod deploy.yml
      -e "app_version=$CI_COMMIT_SHA"
      -e "app_image=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
  environment:
    name: production
    url: https://example.com
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual
  allow_failure: false

Advanced pipelines

GitLab offers advanced features to build high-performance and flexible pipelines suited to complex projects.

Rules: execution conditions

The rules keyword replaces only/except and offers more powerful conditional logic:

deploy_staging:
  stage: staging
  script:
    - ./deploy.sh staging
  rules:
    # Deploy to develop automatically
    - if: $CI_COMMIT_BRANCH == "develop"
      when: always
    # Deploy manually on MRs
    - if: $CI_MERGE_REQUEST_IID
      when: manual
    # Do not run in other cases
    - when: never

Needs: dependency graph (DAG)

The needs keyword lets a job start as soon as its dependencies are finished, without waiting for the entire stage to complete:

stages:
  - build
  - test
  - deploy

build_frontend:
  stage: build
  script: npm run build:frontend

build_backend:
  stage: build
  script: go build ./...

test_frontend:
  stage: test
  needs: ["build_frontend"]  # Starts as soon as build_frontend is done
  script: npm run test:frontend

test_backend:
  stage: test
  needs: ["build_backend"]   # Starts as soon as build_backend is done
  script: go test ./...

deploy:
  stage: deploy
  needs: ["test_frontend", "test_backend"]
  script: ./deploy.sh

Parallel jobs

test_parallel:
  stage: test
  parallel: 4
  script:
    - echo "Running job $CI_NODE_INDEX of $CI_NODE_TOTAL"
    - npm run test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

# Multi-version test matrix
test_matrix:
  stage: test
  parallel:
    matrix:
      - PYTHON_VERSION: ["3.10", "3.11", "3.12"]
        DB: ["postgres", "mysql"]
  image: python:${PYTHON_VERSION}-slim
  services:
    - $DB
  script:
    - pip install -r requirements.txt
    - pytest tests/

Child pipelines (include/trigger)

# Parent pipeline
stages:
  - triggers

frontend:
  stage: triggers
  trigger:
    include: frontend/.gitlab-ci.yml
    strategy: depend
  rules:
    - changes:
        - frontend/**/*

backend:
  stage: triggers
  trigger:
    include: backend/.gitlab-ci.yml
    strategy: depend
  rules:
    - changes:
        - backend/**/*

Securing the pipeline

GitLab integrates security analysis tools directly into the CI/CD pipeline. These scanners detect vulnerabilities before going to production.

SAST (Static Application Security Testing)

include:
  - template: Security/SAST.gitlab-ci.yml

# SAST customization
sast:
  stage: test
  variables:
    SAST_EXCLUDED_PATHS: "spec,test,tests,docs"
    SEARCH_MAX_DEPTH: 4

Dependency Scanning

include:
  - template: Security/Dependency-Scanning.gitlab-ci.yml

dependency_scanning:
  stage: test
  variables:
    DS_EXCLUDED_ANALYZERS: "gemnasium-python"

DAST (Dynamic Application Security Testing)

include:
  - template: Security/DAST.gitlab-ci.yml

dast:
  stage: staging
  variables:
    DAST_WEBSITE: "https://staging.example.com"
    DAST_FULL_SCAN_ENABLED: "true"
  rules:
    - if: $CI_COMMIT_BRANCH == "develop"

Complete security pipeline

include:
  - template: Security/SAST.gitlab-ci.yml
  - template: Security/Dependency-Scanning.gitlab-ci.yml
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml
  - template: Security/License-Scanning.gitlab-ci.yml

# The results appear in
# Security & Compliance > Vulnerability Report

Troubleshooting

Here are the most common problems encountered with GitLab CI/CD and their solutions.

Pipeline does not start

# Check the syntax of the YAML file
# In GitLab: CI/CD > Pipelines > CI Lint
# Or locally with:
pip install gitlab-ci-lint
gitlab-ci-lint .gitlab-ci.yml

• Check the rules/only/except: The job may be excluded by a condition
• Protected branch: Runners may only process protected branches
• Available runners: In Settings > CI/CD > Runners, verify that active runners exist

Job fails with a permission error

# Add the necessary permissions
job_with_perms:
  script:
    - chmod +x scripts/deploy.sh
    - ./scripts/deploy.sh
  # Or use before_script to install tools
  before_script:
    - apt-get update && apt-get install -y curl jq

Cache not available

• The cache is best-effort: it may not be available after a runner change
• Check the cache key: use key: files for a cache based on file contents
• Docker runners use a volume for the cache; make sure it is configured correctly

Debug mode

# Enable debug mode for a job
debug_job:
  variables:
    CI_DEBUG_TRACE: "true"
  script:
    - echo "The executed commands are displayed in detail"
    - env | sort   # List all environment variables

Conclusion

GitLab CI/CD is a complete and mature tool that covers the entire DevOps lifecycle, from continuous integration to production deployment. By mastering the concepts covered in this guide, you can now:

• Configure robust multi-stage pipelines with the .gitlab-ci.yml file
• Install and manage GitLab Runners tailored to your needs
• Protect your secrets with protected and masked CI/CD variables
• Optimize your build speed with the cache and artifacts
• Build Docker images securely with Kaniko
• Automate your tests and track code coverage
• Deploy with environments, review apps and manual approval
• Integrate security analyses (SAST, DAST, dependency scanning)
• Create advanced pipelines with rules, needs and child pipelines

Gradually adopting these practices will transform your development workflow: every commit is automatically built, tested, analyzed and ready to be deployed with full confidence.