An observability baseline for platforms and services
A shared metrics, logs and alerting stack with defaults every service inherits, so problems are found by alerts rather than by users.
Problem
Monitoring was inconsistent. Some services had dashboards, few had useful alerts, and logs lived on the machines that produced them.
Constraints
- Self-hosted; data could not leave the organization's infrastructure.
- Alerts had to be actionable, or they would be ignored.
Decisions
- Prometheus and Grafana for metrics, Loki for logs
- Open, self-hostable, Kubernetes-native, and one query experience across metrics and logs.
- Alert on symptoms users feel, not every internal cause
- Fewer, better alerts. Causes are investigated from dashboards once a symptom alert fires.
- Dashboards and alert rules stored in Git
- Observability changes are reviewed and reproducible, like the rest of the platform.
Architecture
Collection
Exporters and service metrics scraped; container and system logs shipped by node agents.
- Prometheus
- node_exporter
- Promtail or Alloy
Storage
Metrics and logs retained centrally with defined retention per environment.
- Prometheus
- Loki
Visualization
Shared dashboards per platform layer, plus a standard dashboard every service gets.
- Grafana
Alerting
Alert rules routed by severity and ownership to the right team.
- Alertmanager
Build notes
Implementation
The stack was deployed to the Kubernetes platform through the same GitOps flow as everything else. Every new service received a default dashboard and a small set of golden-signal alerts automatically, which teams could extend.
Outcome
Platform and service health is visible in one place, logs are searchable without logging into servers, and alerts go to the people who own the system.
What I learned
An alert nobody acts on is worse than no alert. Every alert should say who owns it and what to do first.