Custom metrics with OpenTelemetry

Auto-instrumentation gives you request rate, error rate, and latency for free. Custom metrics are for the numbers that are specific to your application — payments processed, items in a queue, cache hit ratio, model inference time. You define them with the OpenTelemetry metrics API and they arrive in Osuite over the same OTLP exporter as your traces and logs.

Note

These examples assume the OpenTelemetry SDK is already initialized in your service, with a metric reader and an OTLP exporter configured. If it is not, set that up first — the Instrumentation Agent or the manual instrumentation guides wire up the SDK and the exporter. This page only adds instruments on top of that setup.

The three instrument types

You will use three instruments, one per metric type:

• Counter — a value you add to as events happen (add(1)). Only goes up.
• Histogram — a value you record per event (record(value)); the SDK buckets the distribution so you can compute percentiles later.
• Observable gauge — a value the SDK reads on each export by calling a callback you provide. Use it for a current level you can measure on demand, such as queue depth or connection count.

Define and record custom metrics

Get a meter, create your instruments once at startup, then record against them from your application code.

Node.js

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('payment-service');

const paymentsProcessed = meter.createCounter('payments.processed', {
  description: 'Number of payments processed',
});

const paymentAmount = meter.createHistogram('payment.amount', {
  description: 'Distribution of payment amounts',
  unit: 'USD',
});

const queueDepth = meter.createObservableGauge('payment.queue.depth', {
  description: 'Payments awaiting processing',
});
queueDepth.addCallback((result) => {
  result.observe(getQueueDepth());
});

export function onPayment(amount, currency) {
  paymentsProcessed.add(1, { currency });
  paymentAmount.record(amount, { currency });
}

Python

from opentelemetry import metrics
from opentelemetry.metrics import CallbackOptions, Observation

meter = metrics.get_meter("payment-service")

payments_processed = meter.create_counter(
    "payments.processed",
    description="Number of payments processed",
)

payment_amount = meter.create_histogram(
    "payment.amount",
    unit="USD",
    description="Distribution of payment amounts",
)

def observe_queue_depth(options: CallbackOptions):
    yield Observation(get_queue_depth())

meter.create_observable_gauge(
    "payment.queue.depth",
    callbacks=[observe_queue_depth],
    description="Payments awaiting processing",
)

def on_payment(amount, currency):
    payments_processed.add(1, {"currency": currency})
    payment_amount.record(amount, {"currency": currency})

Go

package payments

import (
  "context"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/metric"
)

var meter = otel.Meter("payment-service")

var (
  paymentsProcessed, _ = meter.Int64Counter(
    "payments.processed",
    metric.WithDescription("Number of payments processed"),
  )
  paymentAmount, _ = meter.Float64Histogram(
    "payment.amount",
    metric.WithUnit("USD"),
    metric.WithDescription("Distribution of payment amounts"),
  )
)

func init() {
  queueDepth, _ := meter.Int64ObservableGauge(
    "payment.queue.depth",
    metric.WithDescription("Payments awaiting processing"),
  )
  _, _ = meter.RegisterCallback(
    func(ctx context.Context, o metric.Observer) error {
      o.ObserveInt64(queueDepth, getQueueDepth())
      return nil
    },
    queueDepth,
  )
}

func OnPayment(ctx context.Context, amount float64, currency string) {
  attrs := metric.WithAttributes(attribute.String("currency", currency))
  paymentsProcessed.Add(ctx, 1, attrs)
  paymentAmount.Record(ctx, amount, attrs)
}

Java

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.metrics.DoubleHistogram;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.Meter;

public class PaymentMetrics {
  static final AttributeKey<String> CURRENCY = AttributeKey.stringKey("currency");

  final LongCounter paymentsProcessed;
  final DoubleHistogram paymentAmount;

  public PaymentMetrics(PaymentQueue queue) {
    Meter meter = GlobalOpenTelemetry.getMeter("payment-service");

    paymentsProcessed = meter
        .counterBuilder("payments.processed")
        .setDescription("Number of payments processed")
        .build();

    paymentAmount = meter
        .histogramBuilder("payment.amount")
        .setUnit("USD")
        .setDescription("Distribution of payment amounts")
        .build();

    meter.gaugeBuilder("payment.queue.depth")
        .setDescription("Payments awaiting processing")
        .buildWithCallback(m -> m.record(queue.depth()));
  }

  public void onPayment(double amount, String currency) {
    Attributes attrs = Attributes.of(CURRENCY, currency);
    paymentsProcessed.add(1, attrs);
    paymentAmount.record(amount, attrs);
  }
}

.NET

using System.Collections.Generic;
using System.Diagnostics.Metrics;

public class PaymentMetrics
{
    public static readonly Meter Meter = new("payment-service");

    private readonly Counter<long> _paymentsProcessed;
    private readonly Histogram<double> _paymentAmount;

    public PaymentMetrics(PaymentQueue queue)
    {
        _paymentsProcessed = Meter.CreateCounter<long>(
            "payments.processed",
            description: "Number of payments processed");

        _paymentAmount = Meter.CreateHistogram<double>(
            "payment.amount",
            unit: "USD",
            description: "Distribution of payment amounts");

        Meter.CreateObservableGauge<long>(
            "payment.queue.depth",
            () => queue.Depth,
            description: "Payments awaiting processing");
    }

    public void OnPayment(double amount, string currency)
    {
        var tag = new KeyValuePair<string, object?>("currency", currency);
        _paymentsProcessed.Add(1, tag);
        _paymentAmount.Record(amount, tag);
    }
}

Caution

In .NET the Meter name must be registered with the MeterProvider or its metrics are never exported. Add .AddMeter("payment-service") to your MeterProviderBuilder (or set the OTEL_DOTNET_AUTO_METRICS_ADDITIONAL_SOURCES environment variable when using the auto-instrumentation agent).

Send them to Osuite

Custom metrics use the same OTLP exporter as the rest of your telemetry. Set these environment variables wherever the service runs — they configure the exporter for traces, logs, and metrics alike:

OTEL_SERVICE_NAME="payment-service"
OTEL_RESOURCE_ATTRIBUTES="service.environment=production"
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.osuite.io:443"
OTEL_EXPORTER_OTLP_HEADERS="x-osuite-ingest-token=<your-ingest-token>"

service.environment is the key Osuite reads to separate telemetry across production, staging, and development — set it per environment in your deploy. Use service.environment, not the standard OTel deployment.environment, which Osuite does not read.

How custom metrics appear in Osuite

Osuite stores metrics under Prometheus-normalized names, so the name you query is not always the name you declared:

• Dots become underscores: payments.processed is queried as payments_processed.
• Counters gain a _total suffix: payments.processed becomes payments_processed_total.
• A histogram named payment.amount becomes three series — payment_amount_bucket, payment_amount_sum, and payment_amount_count.

See Querying with PromQL for how to use each of these.

Verify in Osuite

What you should see

Within a minute of your service recording its first data point, the new metrics appear in the Explore metrics view under their normalized names, with the currency label available to filter and group by.

Troubleshooting

Custom metrics are only emitted while your application is calling add / record (synchronous instruments) or while the export callback is firing (observable gauge). A metric that is never exercised will not appear.

No data in Osuite after a few minutes? Work through these checks.

• Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
• Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
• Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
• Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
• Network — confirm the host has outbound access to the ingest endpoint on port 443.
• Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.

Next steps

• Querying with PromQL — query your new metrics
• Prometheus scraping — collect metrics from existing /metrics endpoints instead
• Alerts — alert on a custom metric