Skip to content
Osuite Osuite Docs

PromQL alert examples

These are copy-paste PromQL recipes for the alerts you set up most often. Each one gives the query, a suggested threshold and duration, and the condition that makes it fire. Paste the query into an alert’s query A, set the threshold expression, and set the “for” duration — see Creating an alert for the full flow.

Each recipe uses the metric names Osuite stores. HTTP metrics come from OpenTelemetry SDK instrumentation, host metrics from the hostmetrics receiver, and Kubernetes metrics from kube-state-metrics and the kubelet via the osuite-k8s chart. Replace the example service, host_name, and namespace label values with your own.

Error rate

Section titled “Error rate”

Fires on the fraction of requests that return a 5xx response, using the http_server_request_duration_seconds_count counter emitted by instrumented HTTP servers.

sum(rate(http_server_request_duration_seconds_count{service="payment-service", http_response_status_code=~"5.."}[5m]))
/
sum(rate(http_server_request_duration_seconds_count{service="payment-service"}[5m]))
  • Threshold: A > 0.05
  • For: 5m

Fires when more than 5% of requests to payment-service return a 5xx status continuously for five minutes. The for window ignores momentary spikes from a single failed request or a deploy blip.

Latency

Section titled “Latency”

Both recipes read a percentile from the http_server_request_duration_seconds_bucket histogram with histogram_quantile(). The le label is the bucket boundary and must survive the aggregation, so it is grouped alongside service. The value is in seconds.

p95 latency

Section titled “p95 latency”
histogram_quantile(
  0.95,
  sum by (le, service) (rate(http_server_request_duration_seconds_bucket{service="payment-service"}[5m]))
)
  • Threshold: A > 0.5
  • For: 10m

Fires when the 95th-percentile request duration for payment-service stays above 500 ms for ten minutes — a sustained slowdown affecting a meaningful share of traffic.

p99 latency

Section titled “p99 latency”
histogram_quantile(
  0.99,
  sum by (le, service) (rate(http_server_request_duration_seconds_bucket{service="payment-service"}[5m]))
)
  • Threshold: A > 1
  • For: 10m

Fires when the 99th-percentile request duration exceeds one second for ten minutes, catching the slow tail even when the median looks healthy.

Host resources

Section titled “Host resources”

Host metrics come from the OpenTelemetry hostmetrics receiver. The utilization metrics are gauges reporting a fraction between 0 and 1, broken out by a state label, and are grouped here by the host_name resource attribute.

High CPU

Section titled “High CPU”
1 - avg by (host_name) (system_cpu_utilization{state="idle"})
  • Threshold: A > 0.9
  • For: 10m

Fires when a host’s non-idle CPU averages above 90% for ten minutes. Subtracting idle from 1 gives total busy time across user, system, and wait states.

High memory

Section titled “High memory”
avg by (host_name) (system_memory_utilization{state="used"})
  • Threshold: A > 0.9
  • For: 10m

Fires when more than 90% of a host’s memory is in the used state for ten minutes, well before the host starts swapping or the OOM killer engages.

Low disk space

Section titled “Low disk space”
max by (host_name, mountpoint) (system_filesystem_utilization{state="used"})
  • Threshold: A > 0.85
  • For: 15m

Fires when any mounted filesystem on a host is more than 85% full for fifteen minutes. Grouping by mountpoint keeps a full data volume from being averaged away by empty ones.

Kubernetes

Section titled “Kubernetes”

Kubernetes metrics come from kube-state-metrics (kube_* series) and the kubelet and cAdvisor (container_* series), both collected by the osuite-k8s chart.

Pod restarts

Section titled “Pod restarts”
sum by (namespace, pod) (increase(kube_pod_container_status_restarts_total[15m]))
  • Threshold: A > 3
  • For: 5m

Fires when a pod’s containers restart more than three times in fifteen minutes — the signature of a crash loop rather than a single unlucky restart.

Container memory near limit

Section titled “Container memory near limit”
sum by (namespace, pod) (container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory"})
  • Threshold: A > 0.9
  • For: 10m

Fires when a pod’s working-set memory exceeds 90% of its configured memory limit for ten minutes, warning of an imminent OOMKill before the container is terminated.

Node pressure

Section titled “Node pressure”
max by (node) (kube_node_status_condition{condition="MemoryPressure", status="true"})
  • Threshold: A >= 1
  • For: 5m

Fires when a node reports the MemoryPressure condition for five minutes, meaning the kubelet is under memory pressure and may start evicting pods. Swap condition for DiskPressure or PIDPressure to watch the other node conditions.

Next steps

Section titled “Next steps”