Alerts

Alerts in Osuite watch your metrics and notify you when a condition holds. Every alert is defined by a PromQL expression evaluated on a fixed interval: when the expression crosses the threshold you set and stays there for the duration you set, the alert fires and notifications go out.

Alerts are PromQL-only. They evaluate metric queries — there are no log-based alerts. To alert on something a log records, expose it as a metric first and query that metric.

Tip

You do not have to write the PromQL by hand. The query-writing agent composes a correct PromQL expression from a plain-English description of the condition you want to alert on, working against your real metric catalog so it never invents a metric or label.

Creating an alert

Open Alerts → New Alert in the Osuite UI.

1. Write the PromQL query. An alert has one or more queries, each labelled with a letter (A, B, …). Query A is the metric you want to watch. For example, the ratio of 5xx responses to all requests for a service:

   sum(rate(http_server_request_duration_seconds_count{service="payment-service", http_response_status_code=~"5.."}[5m]))
   /
   sum(rate(http_server_request_duration_seconds_count{service="payment-service"}[5m]))

2. Set the threshold expression. The threshold is an expression over your query letters that evaluates to true or false. To fire when the error ratio above exceeds 5%:

   A > 0.05

   When the expression references more than one query, add the other queries (B, C, …) and combine them here — for example A / B > 0.05.

3. Set the evaluation interval and duration. The evaluation interval is how often Osuite runs the query. The duration (the “for” window) is how long the threshold expression must stay true before the alert fires. A duration of 5m means a momentary spike is ignored and the alert only fires if the condition holds continuously for five minutes.

4. Assign a severity. Severity tells whoever is on call how urgently to respond:

   • Critical — production is down or severely degraded
   • Warning — something is off but not yet user-impacting
   • Info — an informational threshold for awareness

5. Choose notification channels. Select which of your configured channels the alert should notify when it fires. See Notification channels to connect Slack, Opsgenie, Zenduty, or PagerDuty first.

6. Name and save the alert. Give it a name that describes the impact (High error rate – payment-service), then save.

What you should see

The new alert appears in Alerts → All Alerts in the Normal state. When you save an alert whose condition is already true, it moves to Pending and then Firing after its duration elapses.

Alert states

An alert is always in exactly one of three states. The difference between Pending and Firing is the duration window: an alert is Pending while its condition is met but has not yet held long enough, and Firing once it has.

State	What it means	Notifies?
Normal	The threshold expression evaluates to false. The condition is not met.	No
Pending	The condition is met, but has not yet held for the full duration window.	No
Firing	The condition has held continuously for the full duration window.	Yes — notifications are sent to the selected channels

An alert returns to Normal as soon as its expression evaluates to false again, and notifies the same channels that it has resolved.

Managing alerts

Alert list

Alerts → All Alerts lists every alert with its current state, severity, and the channels it notifies. Select an alert to see its query, threshold, and recent state history.

Muting alerts

Mute an alert for a defined window when you expect the condition to be true — planned maintenance, a deployment, or a load test. Open the alert and click Pause; it keeps evaluating but sends no notifications until the window ends.

Next steps

• Notification channels — connect Slack, Opsgenie, Zenduty, and PagerDuty
• Alert examples — ready-to-use PromQL recipes for error rate, latency, hosts, and Kubernetes
• Best practices — symptom-based alerting, severity, and avoiding noisy alerts