Prometheus + Grafana: Complete Monitoring Stack

Key takeaways

• The stack is four components on four ports: Prometheus (9090), which fetches metrics itself over HTTP from each target's /metrics endpoint (pull model), node_exporter (9100), which exposes system metrics, Grafana (3000) for visualization and Alertmanager (9093), which deduplicates, groups and routes alerts.
• Always validate before restarting: promtool check config /etc/prometheus/prometheus.yml, promtool check rules for alerting rules and amtool check-config on the Alertmanager side. With the --web.enable-lifecycle flag, a simple curl -X POST http://localhost:9090/-/reload hot-reloads the configuration without interrupting scraping.
• Three PromQL queries cover most needs: CPU 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100), memory (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 and latency histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) — rate() only applies to counters. In Grafana, importing the Node Exporter Full dashboard (ID 1860) gives you those views without writing a single query.
• An alerting rule is only as good as its for: clause, which filters out transient spikes: up == 0 for 2 minutes for a dead instance, CPU > 85 % for 5 minutes, disk > 85 % for 10 minutes. In Alertmanager, group_wait: 30s, group_interval: 5m and repeat_interval: 4h (1 h for critical) keep the noise down, and inhibit_rules mute warnings as soon as a critical alert exists for the same alertname/instance pair.
• Never expose ports 9090, 9100 and 9093 to the Internet. Prometheus has supported basic authentication natively since version 2.24 through --web.config.file and a bcrypt hash generated with htpasswd -nBC 10; otherwise put it behind an Nginx reverse proxy with TLS. On the Grafana side, change the default admin/admin and set allow_sign_up = false.

+------------------+       +-------------------+       +------------------+
|   Node Exporter  | <---- |    Prometheus      | ----> |   Alertmanager   |
|   (port 9100)    |       |    (port 9090)     |       |   (port 9093)    |
+------------------+       +-------------------+       +------------------+
                                    |
                                    v
                           +-------------------+
                           |     Grafana        |
                           |    (port 3000)     |
                           +-------------------+