systemd: Complete Linux Service Management

Prerequisites

• Operating system: Linux distribution using systemd (Debian 8+, Ubuntu 15.04+, CentOS 7+, Fedora 15+)
• Privileges: root access or sudo privileges for service management
• Knowledge: Basics of Linux administration and the terminal
• Version: systemd 245+ recommended for advanced features

Check your systemd version:

systemctl --version
# Output: systemd 252 (252.22-1~deb12u1)

systemd architecture

PID 1: the initial process

systemd is the first process launched by the Linux kernel (PID 1). It is responsible for starting all other processes and services on the system. Unlike SysVinit, which ran scripts sequentially, systemd parallelizes startup thanks to its dependency system.

Units

The basic building block of systemd is the unit. Every resource managed by systemd is represented by a unit file. There are several types:

• .service: a daemon or a process (nginx, sshd, postgresql)
• .socket: a network or IPC socket for on-demand activation
• .timer: a timer that triggers a service (an alternative to cron)
• .target: a group of units (equivalent to runlevels)
• .mount / .automount: a filesystem mount point
• .path: file change monitoring
• .slice: a resource control group (cgroups)

Unit file locations

# Units installed by packages (do not modify)
/usr/lib/systemd/system/

# Custom units and overrides (take priority)
/etc/systemd/system/

# Temporary units created at runtime
/run/systemd/system/

# List all available unit files
systemctl list-unit-files

Dependencies and startup order

systemd automatically resolves dependencies between units. The Requires=, Wants=, After= and Before= directives define the relationships:

# Visualize a service's dependencies
systemctl list-dependencies nginx.service

# Show the reverse tree (what depends on this service)
systemctl list-dependencies --reverse nginx.service

Service management

Essential commands with systemctl

systemctl is the central command for interacting with systemd. Here are the fundamental operations:

# Start a service
sudo systemctl start nginx

# Stop a service
sudo systemctl stop nginx

# Restart a service (stop + start)
sudo systemctl restart nginx

# Reload the configuration without restarting
sudo systemctl reload nginx

# Restart if active, do nothing otherwise
sudo systemctl try-restart nginx

# Reload or restart if reload is not supported
sudo systemctl reload-or-restart nginx

State and information

# Detailed state of a service
systemctl status nginx

# Check whether a service is active
systemctl is-active nginx

# Check whether a service is enabled at boot
systemctl is-enabled nginx

# Check whether a service has failed
systemctl is-failed nginx

# List all active services
systemctl list-units --type=service --state=running

# List failed services
systemctl --failed

Enabling at startup

# Enable a service at boot
sudo systemctl enable nginx

# Disable a service at boot
sudo systemctl disable nginx

# Enable AND start in a single command
sudo systemctl enable --now nginx

# Disable AND stop in a single command
sudo systemctl disable --now nginx

# Prevent a service from being started (masking)
sudo systemctl mask nginx

# Undo masking
sudo systemctl unmask nginx

Creating custom services

Structure of a unit file

A .service unit file consists of three main sections:

[Unit]
Description=My Python Web Application
Documentation=https://example.com/docs
After=network.target postgresql.service
Wants=postgresql.service

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/my-app
Environment=ENV=production
EnvironmentFile=/opt/my-app/.env
ExecStartPre=/opt/my-app/venv/bin/python manage.py migrate
ExecStart=/opt/my-app/venv/bin/gunicorn app:application --bind 0.0.0.0:8000
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
SyslogIdentifier=my-app

[Install]
WantedBy=multi-user.target

Section details

• [Unit]: metadata and dependencies. After= controls the startup order, Wants= declares a soft dependency
• [Service]: service behavior. ExecStart= is the main command, Restart= defines the restart policy
• [Install]: activation target. WantedBy=multi-user.target means the service is enabled in standard multi-user mode

Deploying a custom service

# Create the unit file
sudo nano /etc/systemd/system/my-app.service

# Reload the systemd configuration
sudo systemctl daemon-reload

# Enable and start the service
sudo systemctl enable --now my-app

# Verify it is running
systemctl status my-app

Service types

The Type= directive in the [Service] section defines how systemd determines that the service has started successfully:

• simple (default): systemd considers the service started as soon as ExecStart= is launched. Ideal for processes that stay in the foreground.
• forking: the process forks and the parent exits. systemd waits for the parent to terminate. Use PIDFile= to track the child process.
• oneshot: similar to simple, but systemd waits for the process to finish before considering the service active. Ideal for initialization scripts.
• notify: the service sends a signal to systemd via sd_notify() when it is ready. The most reliable for applications that support it.
• exec: similar to simple, but systemd waits until the binary has actually been executed (after exec()) before considering startup successful.

# Example: forking service (classic daemon)
[Service]
Type=forking
PIDFile=/var/run/my-daemon.pid
ExecStart=/usr/sbin/my-daemon --daemon

# Example: oneshot service (configuration script)
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/configure-network.sh
ExecStop=/usr/local/bin/deconfigure-network.sh

systemd timers

systemd timers are a modern and more powerful alternative to cron. Each timer is associated with a service of the same name that it triggers.

Creating a timer

First, create the service that will be triggered:

# /etc/systemd/system/backup-db.service
[Unit]
Description=Daily database backup

[Service]
Type=oneshot
User=postgres
ExecStart=/opt/scripts/backup-database.sh
StandardOutput=journal

Then, create the associated timer:

# /etc/systemd/system/backup-db.timer
[Unit]
Description=Daily backup schedule

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target

OnCalendar syntax

# Format: DayOfWeek Year-Month-Day Hour:Minute:Second
OnCalendar=*-*-* 02:00:00          # Every day at 2 AM
OnCalendar=Mon *-*-* 09:00:00      # Every Monday at 9 AM
OnCalendar=*-*-01 00:00:00         # The 1st of every month
OnCalendar=hourly                   # Every hour
OnCalendar=weekly                   # Every week

# Other time-based triggers
OnBootSec=5min                      # 5 minutes after boot
OnUnitActiveSec=1h                  # 1 hour after the last activation

# Validate a calendar's syntax
systemd-analyze calendar "Mon *-*-* 09:00:00"

Managing timers

# Enable and start the timer
sudo systemctl enable --now backup-db.timer

# List all active timers
systemctl list-timers --all

# Manually trigger the associated service
sudo systemctl start backup-db.service

Journald: centralized log management

journald is the logging system built into systemd. It collects and indexes the logs of all services, the kernel and syslog messages in a structured binary format.

Viewing logs with journalctl

# All system logs
journalctl

# Logs of a specific service
journalctl -u nginx.service

# Follow logs in real time
journalctl -u nginx.service -f

# Logs since the last boot
journalctl -b

# Logs from the previous boot
journalctl -b -1

# Filter by date
journalctl --since "2026-02-12 08:00:00" --until "2026-02-12 12:00:00"
journalctl --since "1 hour ago"

# Filter by priority (0=emerg to 7=debug)
journalctl -p err                   # Errors and above
journalctl -p warning -u nginx      # nginx warnings

# Kernel logs only
journalctl -k

# Custom output format
journalctl -u nginx -o json-pretty  # JSON format
journalctl -u nginx -o short-iso    # ISO timestamps

Log persistence

By default on some distributions, logs do not survive a reboot. To enable persistence:

# Create the persistent storage directory
sudo mkdir -p /var/log/journal

# Restart journald to apply
sudo systemctl restart systemd-journald

Configuration and rotation

The /etc/systemd/journald.conf file controls journald's behavior:

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=500M
SystemMaxFileSize=50M
MaxRetentionSec=1month
ForwardToSyslog=no

# See the disk space used by logs
journalctl --disk-usage

# Clean up old logs (keep 500M)
sudo journalctl --vacuum-size=500M

# Clean up logs older than 2 weeks
sudo journalctl --vacuum-time=2weeks

Targets and the boot process

Targets are groups of units that represent a system state. They replace the old SysVinit runlevels:

• poweroff.target (runlevel 0): system shutdown
• rescue.target (runlevel 1): single-user rescue mode
• multi-user.target (runlevel 3): multi-user mode without a graphical interface
• graphical.target (runlevel 5): full graphical mode
• reboot.target (runlevel 6): reboot
• emergency.target: minimal emergency mode (root shell)

# See the default target
systemctl get-default

# Change the default target
sudo systemctl set-default multi-user.target

# Switch to a target immediately
sudo systemctl isolate rescue.target

# List the units of a target
systemctl list-dependencies multi-user.target

Socket activation

Socket activation lets systemd listen on a port and only start the associated service when a connection arrives. This reduces resource consumption and speeds up boot.

# /etc/systemd/system/my-api.socket
[Unit]
Description=Socket for My API

[Socket]
ListenStream=8080
Accept=no
BindIPv6Only=both

[Install]
WantedBy=sockets.target

# /etc/systemd/system/my-api.service
[Unit]
Description=My REST API
Requires=my-api.socket

[Service]
Type=simple
User=apiuser
ExecStart=/opt/api/bin/server
StandardOutput=journal

[Install]
WantedBy=multi-user.target

# Enable the socket (not the service directly)
sudo systemctl enable --now my-api.socket

# The service starts automatically on the first connection
curl http://localhost:8080/health

Resource control

systemd integrates with the Linux kernel's cgroups v2 to limit the resources consumed by each service:

# In the [Service] section of a unit file
[Service]
# Limit CPU to 50%
CPUQuota=50%

# Limit memory to 512 MB (hard limit)
MemoryMax=512M

# Soft memory limit (triggers swap)
MemoryHigh=400M

# Limit disk I/O
IOWeight=50
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M

# Limit the number of processes
TasksMax=64

# See the resources consumed by a service
systemctl status nginx.service     # Quick overview
systemd-cgtop                      # Real-time top-style view

# Apply limits on the fly (temporary)
sudo systemctl set-property nginx.service CPUQuota=30%
sudo systemctl set-property nginx.service MemoryMax=256M

# See a service's resource properties
systemctl show nginx.service -p CPUQuota,MemoryMax,TasksMax

Analysis and debugging

systemd-analyze provides powerful tools to understand and optimize the startup process:

# Total startup time
systemd-analyze time

# Startup time per service (sorted by duration)
systemd-analyze blame

# Critical chain: the longest path of the boot
systemd-analyze critical-chain

# Critical chain of a specific service
systemd-analyze critical-chain nginx.service

# Generate an SVG chart of the boot
systemd-analyze plot > boot-chart.svg

# Check the syntax of a unit file
systemd-analyze verify /etc/systemd/system/my-app.service

# Analyze a service's security score
systemd-analyze security nginx.service

# Analyze the security of all services
systemd-analyze security

Service hardening

systemd offers many sandboxing directives to isolate services and reduce the attack surface. Here are the most important options:

[Service]
# Filesystem protection
ProtectSystem=strict           # / read-only except /dev, /proc, /sys
ReadWritePaths=/var/lib/my-app /var/log/my-app
ProtectHome=true               # /home, /root, /run/user inaccessible
PrivateTmp=true                # isolated /tmp for this service

# Network and device isolation
PrivateDevices=true            # No access to physical devices
ProtectKernelTunables=true     # /proc/sys read-only
ProtectKernelModules=true      # Loading modules impossible
ProtectControlGroups=true      # /sys/fs/cgroup read-only

# Privilege restriction
NoNewPrivileges=true           # Prevents privilege escalation
CapabilityBoundingSet=CAP_NET_BIND_SERVICE  # Only allowed capabilities
AmbientCapabilities=CAP_NET_BIND_SERVICE

# System call filtering
SystemCallFilter=@system-service  # Only allows standard syscalls
SystemCallArchitectures=native    # Blocks syscalls from other architectures

# Additional restrictions
RestrictNamespaces=true        # No namespace creation
RestrictRealtime=true          # No real-time scheduling
RestrictSUIDSGID=true          # No SUID/SGID files
LockPersonality=true           # Locks the execution domain
MemoryDenyWriteExecute=true    # Prevents W^X (self-modifying code)

# Evaluate a service's current security level
systemd-analyze security nginx.service

# Output: EXPOSURE LEVEL with a score from 0 (safe) to 10 (unsafe)
# The goal is to achieve a score below 5

Troubleshooting

Diagnosing a failed service

# 1. Check the state and the latest log lines
systemctl status my-app.service

# 2. View the full logs with context
journalctl -xeu my-app.service

# 3. Check the unit file syntax
systemd-analyze verify /etc/systemd/system/my-app.service

# 4. Test the command manually
sudo -u appuser /opt/my-app/venv/bin/gunicorn app:application

# 5. Check permissions
ls -la /opt/my-app/
namei -l /opt/my-app/venv/bin/gunicorn

Common problems and solutions

• Service in a restart loop: Check StartLimitBurst and StartLimitIntervalSec in the [Unit] section. Use systemctl reset-failed to reset the counter.
• Forgotten "daemon-reload": After any modification to a unit file, run sudo systemctl daemon-reload so systemd takes the changes into account.
• Insufficient permissions: Make sure the user defined by User= has the necessary rights on the files and directories.
• Missing dependency: If a service fails to start because of another, check the After= and Requires= directives.
• Conflict with an override: Check overrides with systemctl cat my-app.service, which displays the complete effective configuration.

# Reset a failed service
sudo systemctl reset-failed my-app.service

# See the effective configuration (file + overrides)
systemctl cat my-app.service

# Reload after modification
sudo systemctl daemon-reload
sudo systemctl restart my-app.service

Conclusion

systemd is far more than a simple service manager: it is a complete system administration ecosystem. By mastering its components, you have the tools to:

• Efficiently manage the lifecycle of all your services with systemctl
• Create robust custom services with automatic restart and dependency management
• Schedule recurring tasks with timers, more reliable than cron
• Centralize and analyze your logs with journald and journalctl
• Harden your services with built-in sandboxing and reduce the attack surface
• Diagnose and optimize system startup with systemd-analyze
• Finely control the CPU, memory and I/O resources of each service

Mastering systemd is a fundamental skill for any Linux system administrator. Take the time to explore each feature gradually and to consult the manual pages (man systemd.service, man systemd.exec, man journalctl) to dig deeper into each topic.