Querying Osuite metrics with PromQL

Osuite stores metrics as Prometheus-compatible time series and you query them with PromQL — in the Explore metrics view, on dashboard panels, and in alerts. This page covers the parts of PromQL you need for everyday observability work.

Tip

You do not have to write PromQL by hand. The Write PromQL Agent turns a plain-language request into a query built against your real metric catalog, picking the right function for the metric’s type. This page is for when you want to read, adjust, or write the query yourself.

Selectors

A query starts by selecting a metric by name, optionally filtered by labels in braces:

http_server_request_duration_seconds_count{service="payment-service", http_response_status_code="500"}

Label matchers:

• = — exact match
• != — not equal
• =~ — regex match (for example http_response_status_code=~"5.." for all 5xx)
• !~ — regex non-match

A bare selector returns an instant vector — the latest value per series. Append a range in brackets to get a range vector — every sample over that window — which is what rate-style functions consume:

http_server_request_duration_seconds_count[5m]

For ad-hoc queries use a concrete window like [5m] or [1h]. On a dashboard, use the $__rate_interval variable so the window scales with the panel’s zoom.

rate() and counters

A counter only ever increases, so its raw value is not useful on its own. Wrap it in rate() to get the per-second average increase over a window — the standard way to turn a counter into a throughput or error rate:

rate(http_server_request_duration_seconds_count[5m])

• rate() — average per-second rate over the range; use this in almost all cases.
• irate() — instantaneous rate from the last two samples; reacts faster, noisier.
• increase() — total increase over the range (rate() × window), for “how many in the last hour” questions.

Counters instrumented over OTLP end in _total (see naming).

Aggregations

Aggregation operators collapse many series into fewer. By default they collapse everything into one; use by (...) to keep the labels you care about, or without (...) to drop the ones you do not:

sum by (service) (rate(http_server_request_duration_seconds_count[5m]))

• sum, avg, min, max, count — the common reducers.
• by (label, ...) — group the result by these labels; everything else is summed away.
• topk(5, ...) / bottomk(5, ...) — the highest or lowest N series, for “top 5 noisiest endpoints”.

Rate first, then aggregate. sum(rate(...)) is correct; rate(sum(...)) is not.

histogram_quantile()

A histogram exports a set of _bucket series. To read a percentile, take the per-second rate() of the buckets, then apply histogram_quantile() with the quantile you want (0.99 for p99):

histogram_quantile(
  0.99,
  sum by (le) (rate(http_server_request_duration_seconds_bucket[5m]))
)

The le (“less than or equal”) label is the bucket boundary and must be preserved through the aggregation — that is why it is the one label kept in sum by (le). To get a percentile per service, group by both: sum by (le, service).

Common queries

Ready-to-adapt expressions. Replace the metric and label names with your own.

• Request throughput, per service

  sum by (service) (rate(http_server_request_duration_seconds_count[5m]))

• Error rate (fraction of 5xx), per service

  sum by (service) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
  /
  sum by (service) (rate(http_server_request_duration_seconds_count[5m]))

• p95 request latency, per service

  histogram_quantile(
    0.95,
    sum by (le, service) (rate(http_server_request_duration_seconds_bucket[5m]))
  )

• Memory in use, per instance

  avg by (instance) (process_resident_memory_bytes)

• Top 5 endpoints by request volume

  topk(5, sum by (http_route) (rate(http_server_request_duration_seconds_count[5m])))

How OTel metric names map to PromQL

Metrics you send over OTLP are normalized to Prometheus names before they are stored, so query the normalized form:

• Dots become underscores: payment.amount is queried as payment_amount.
• Counters gain a _total suffix: payments.processed becomes payments_processed_total.
• A histogram named payment.amount becomes payment_amount_bucket, payment_amount_sum, and payment_amount_count.

Metrics scraped from a Prometheus endpoint already use Prometheus names, so normalization leaves them effectively unchanged. When in doubt, look the exact name up in the Explore metrics view.

Next steps

• Alerts — every alert is a PromQL expression with a threshold
• Write PromQL Agent — generate a query against your real metrics
• Custom metrics — instrument the metric you want to query