Prometheus · Grafana
Prometheus · Grafana: a first setup, important settings, verification, and recovery considerations.
Working directory: ~/homelab/ops/monitoring. Follow the common setup workflow first. This is a setup note based on the linked documentation, not a copy of my live configuration or a runtime verification of this full stack.
Start here
Start with the Prometheus service already included in the practice lab. It scrapes itself and Traefik every 15 seconds. This gives you two targets to understand before adding host exporters, dashboards, or alerts. No Docker socket or host filesystem access is needed for this exercise.
Settings and data
Read ops/monitoring/prometheus.yml. Each job names an endpoint reachable from the metrics container. metrics:9090 and gateway:8082 are container-network addresses, not public names. The sample retains one day of metrics in a named volume and publishes its UI only on loopback.
Check it
Open Prometheus through the documented SSH forward and query up. Both targets should return 1 after a scrape. Stop only the practice web service: the gateway can remain up while the page fails. Add an application-level uptime check to cover that distinction.
When something fails
For up=0, read the target error and check name, port, path, and network. Grafana later queries Prometheus as a data source; it does not replace collection. Back up configuration, alert rules, and dashboards. Decide separately whether historical metric data needs backup, and keep infrastructure labels private.
Run this only after the setup above, from its project directory:
curl --fail 'http://127.0.0.1:19090/api/v1/query?query=up'