Prometheus + Grafana monitoring stack
This tutorial covers the complete deployment of a monitoring stack with Prometheus for metrics collection, Grafana for visualization, Node Exporter for system metrics and Alertmanager for notifications. By the end of this guide, you will have a working monitoring infrastructure capable of monitoring your servers and alerting you when something goes wrong.
Prerequisites
Operating system: Debian 12+ or Ubuntu 22.04+ (the commands can be adapted to CentOS/RHEL)
Privileges: Root or sudo access on the monitoring server
Resources: Minimum 2 GB of RAM and 20 GB of disk space (adjust according to the number of targets)
Knowledge: Basics of Linux administration and systemd service management
Stack architecture
Prometheus works on a pull model: it is the Prometheus server that periodically queries the targets to retrieve their metrics. This model has several advantages over push (StatsD/Graphite style):
Prometheus: Central server that scrapes metrics over HTTP, stores them locally as time series and evaluates alerting rules
Exporters: Lightweight agents that expose metrics in the Prometheus format on a /metrics endpoint. The most common is node_exporter for system metrics
Grafana: Visualization interface that connects to Prometheus as a data source to create interactive dashboards
Alertmanager: Component that receives the alerts fired by Prometheus, deduplicates them, groups them and routes them to the right recipients (email, Slack, PagerDuty)
Cybersecurity and Linux administration expert. I share my knowledge through free tutorials and training to help system administrators and developers secure their infrastructures.
What is the difference between Prometheus and Grafana?
Prometheus is a metrics collection and storage system based on a pull model: it periodically queries targets to retrieve their metrics. Grafana is a visualization tool that connects to Prometheus (and other sources) to create interactive dashboards. The two are complementary: Prometheus collects, Grafana displays.
How does Prometheus collect metrics?
Prometheus uses a pull model: it sends HTTP GET requests to a /metrics endpoint exposed by each target. Exporters (node_exporter, blackbox_exporter, etc.) convert system or application metrics into the text format expected by Prometheus. The collection frequency is defined in the scrape_interval of the configuration.
Can Prometheus and Grafana be used in production?
Yes, Prometheus and Grafana are used in production by many companies (SoundCloud, DigitalOcean, CERN). Prometheus is designed for reliability: each instance is autonomous and keeps working even if the network or remote storage is unavailable. For high availability, deploy two identical Prometheus instances and use Thanos or Cortex for long-term storage.
What is PromQL and how do I learn it?
PromQL (Prometheus Query Language) is the query language of Prometheus. It lets you select, filter, aggregate and transform time series. The essential functions are rate() for counters, avg/sum/max for aggregation, and histogram_quantile() for percentiles. The best way to learn it is to experiment in the Prometheus interface or in Grafana Explore.
How do I configure alerts with Prometheus?
Alerts are configured in two steps. First, define alerting rules in Prometheus (YAML files with PromQL expressions and thresholds). Then, configure Alertmanager to route those alerts to the right recipients (email, Slack, PagerDuty). Alertmanager handles grouping, silencing and inhibition of alerts to avoid noise.
Which Prometheus exporters should I use to monitor a Linux server?
The node_exporter is essential: it exposes CPU, memory, disk, network and filesystem metrics. Add the blackbox_exporter to monitor HTTP/TCP/ICMP availability, the mysqld_exporter for MySQL/MariaDB, and the nginx-prometheus-exporter for Nginx. Each exporter is deployed as an independent systemd service.
Comments