Skip to content
Osuite Osuite Docs

Alerting best practices

A short, practical guide to alerts that page the right person at the right time and stay quiet otherwise.

Alert on symptoms, not causes

Section titled “Alert on symptoms, not causes”

Alert on what users feel — error rate, latency, and unavailability — not on every internal cause. A spike in CPU is only worth a page if it degrades a user-facing signal; if it doesn’t, it is noise. Symptom-based alerts catch problems you didn’t predict and produce far fewer false pages.

The most valuable first alert for any service is its error rate. Start with a threshold near 1%, watch it for a few days, then tighten. See Alert examples for ready-to-use PromQL.

Tie thresholds to your SLOs

Section titled “Tie thresholds to your SLOs”

If your objective is 99.9% success, a 0.1% error rate is your budget. Set a Warning below that (for example 0.5%) and a Critical at or above it, so you get time to respond before you breach the objective rather than after.

Use severity consistently

Section titled “Use severity consistently”

Severity should map to the response you expect, and mean the same thing across every alert:

  • Critical — wake someone up; production is down or degrading now.
  • Warning — handle within working hours; something is off but not yet user-impacting.
  • Info — no immediate action; awareness only.

If a Critical alert doesn’t warrant an immediate response, it should be a Warning. Alerts that page but don’t need action are the fastest way to train people to ignore the pager.

Avoid flappy alerts

Section titled “Avoid flappy alerts”

A flappy alert fires and resolves repeatedly as a metric hovers around its threshold, burying real incidents in noise.

  • Set a duration window. Requiring the condition to hold for a few minutes (the “for” window) filters momentary spikes. A 2% error blip for 30 seconds rarely warrants a page.
  • Aggregate over a rate window. Query over [5m] rather than instantaneous values so a single data point can’t trip the alert.
  • Leave headroom between severities. Set Warning and Critical thresholds far enough apart that normal variation doesn’t cross both.
  • Mute during expected disruption. Pause an alert for planned maintenance, deployments, or load tests instead of lowering its threshold permanently.

Make alerts actionable

Section titled “Make alerts actionable”

Every alert should tell the responder what broke and what to do about it.

  • Name for impact. High error rate – payment-service is more useful at 3am than payment_service_errors_alert.
  • Link a runbook. Put the link to the response steps in the alert name or description so it travels with the notification into Slack, Opsgenie, Zenduty, or PagerDuty.
  • From alert to root cause. When an alert fires, hand it to the Investigation Agent to trace the symptom back through logs and traces to the underlying cause.

Next steps

Section titled “Next steps”