Go custom OpenTelemetry instrumentation

Framework middleware traces inbound requests. Custom instrumentation adds the detail inside them: spans around the work that matters, attributes that make traces searchable, and application metrics. You take a tracer and a meter from the global providers the SDK bootstrap already registered — no extra wiring per call site.

Prerequisites

The SDK bootstrap from the Go setup page is in place and setupOTel runs at startup.
A registered TracerProvider, which setupOTel sets via otel.SetTracerProvider.

Get a tracer

Take a named tracer from the global provider. The name identifies the instrumentation scope and is usually your module or package path.

import "go.opentelemetry.io/otel"

var tracer = otel.Tracer("checkout-service")

Create a span

Start a span, pass the returned context to any child work, and always end the span. Deferring span.End() is the safe pattern.

func processOrder(ctx context.Context, orderID string) error {
  ctx, span := tracer.Start(ctx, "processOrder")
  defer span.End()

  return charge(ctx)
}

Start returns a new context carrying the span. Passing that ctx down is what makes child spans and downstream calls nest under this one.

Add attributes

Attributes make spans filterable in the trace explorer. Set them on the active span.

import "go.opentelemetry.io/otel/attribute"

span.SetAttributes(
  attribute.String("order.id", orderID),
  attribute.Int("order.items", items),
  attribute.Bool("order.expedited", false),
)

Record events

Events are timestamped log points on a span — useful for marking a milestone within an operation.

span.AddEvent("inventory reserved")

Record errors and set status

Recording an error attaches its message and stack to the span; setting the status to Error is what marks the span as failed in Osuite.

import "go.opentelemetry.io/otel/codes"

if err := charge(ctx); err != nil {
  span.RecordError(err)
  span.SetStatus(codes.Error, "charge failed")
  return err
}

RecordError alone does not fail the span. Always pair it with SetStatus(codes.Error, ...) so error tracking picks it up.

Custom metrics

Metrics need a MeterProvider. The trace bootstrap does not set one up, so add a metrics exporter alongside it.

go get go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc

Add a setupMetrics function next to setupOTel and call it from main, deferring its shutdown the same way.

package main

import (
  "context"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
  sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

func setupMetrics(ctx context.Context) (func(context.Context) error, error) {
  exporter, err := otlpmetricgrpc.New(ctx)
  if err != nil {
    return nil, err
  }
  mp := sdkmetric.NewMeterProvider(
    sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter)),
  )
  otel.SetMeterProvider(mp)
  return mp.Shutdown, nil
}

Take a meter from the global provider and create instruments once, then use them from your handlers.

import (
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/metric"
)

var meter = otel.Meter("checkout-service")

var ordersProcessed, _ = meter.Int64Counter(
  "orders.processed",
  metric.WithDescription("Number of processed orders"),
  metric.WithUnit("{order}"),
)

var orderLatency, _ = meter.Float64Histogram(
  "order.processing.duration",
  metric.WithDescription("Time to process an order"),
  metric.WithUnit("s"),
)

Record from the request path, passing the request context so exemplars can link back to traces.

ordersProcessed.Add(ctx, 1, metric.WithAttributes(
  attribute.String("order.tier", "standard"),
))
orderLatency.Record(ctx, elapsed.Seconds())

For a value you sample rather than accumulate — a queue depth, a pool size — register an observable gauge with a callback the SDK invokes on each collection.

_, err := meter.Int64ObservableGauge(
  "order.queue.depth",
  metric.WithDescription("Pending orders in the queue"),
  metric.WithInt64Callback(func(ctx context.Context, o metric.Int64Observer) error {
    o.Observe(queue.Len())
    return nil
  }),
)

Verify in Osuite

What you should see

Exercise the instrumented code path. Custom spans appear nested inside their request span in the trace explorer with your attributes and events, and custom metrics show up in the Explore metrics view under their instrument names within a minute.

Troubleshooting

Child spans start their own trace. You did not reassign ctx from tracer.Start, or you passed context.Background() into child work. Thread the returned context through.
Errors recorded but the span is not marked failed. You called RecordError without SetStatus(codes.Error, ...).
Custom metrics never appear. No MeterProvider is set — add setupMetrics and call otel.SetMeterProvider, or the metrics exporter is not configured for your ingest endpoint.

No data in Osuite after a few minutes? Work through these checks.

Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
Network — confirm the host has outbound access to the ingest endpoint on port 443.
Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.