gRPC OpenTelemetry instrumentation

Instrument a Go gRPC server and client with OpenTelemetry. The otelgrpc stats handlers turn every unary and streaming RPC into a span and carry trace context across the client-server boundary through gRPC metadata.

Fastest path. Run this in your AI IDE and the Instrumentation Agent wires this service up for you.

/osuite-instrument grpc

It adds the SDK bootstrap and attaches the otelgrpc stats handlers to your server and clients.

Prefer to do it by hand? Follow the manual steps below — they are the complete path.

Prerequisites

Go 1.22 or newer.
A Go module using google.golang.org/grpc.
An Osuite ingest endpoint and token for your region.

Instrument the server

Install the packages.

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
go get google.golang.org/grpc

Create otel.go next to your main package. It initialises the TracerProvider, the propagator, and the service resource, and returns a shutdown function.

package main

import (
  "context"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
  "go.opentelemetry.io/otel/propagation"
  "go.opentelemetry.io/otel/sdk/resource"
  sdktrace "go.opentelemetry.io/otel/sdk/trace"
  semconv "go.opentelemetry.io/otel/semconv/v1.37.0"
)

func setupOTel(ctx context.Context, serviceName, environment string) (func(context.Context) error, error) {
  res, err := resource.New(ctx,
    resource.WithAttributes(
      semconv.ServiceName(serviceName),
      attribute.String("service.environment", environment),
    ),
  )
  if err != nil {
    return nil, err
  }

  exporter, err := otlptracegrpc.New(ctx)
  if err != nil {
    return nil, err
  }

  tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exporter),
    sdktrace.WithResource(res),
  )
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
  ))

  return tp.Shutdown, nil
}

Attach the server stats handler in main. Call setupOTel first, defer the shutdown, then pass otelgrpc.NewServerHandler() to grpc.NewServer. Register your generated service implementation on server before calling Serve.

package main

import (
  "context"
  "log"
  "net"

  "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
  "google.golang.org/grpc"
)

func main() {
  ctx := context.Background()
  shutdown, err := setupOTel(ctx, "checkout-service", "production")
  if err != nil {
    log.Fatalf("otel setup: %v", err)
  }
  defer func() {
    if err := shutdown(context.Background()); err != nil {
      log.Printf("otel shutdown: %v", err)
    }
  }()

  server := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
  )

  lis, err := net.Listen("tcp", ":9090")
  if err != nil {
    log.Fatal(err)
  }
  log.Println("listening on :9090")
  if err := server.Serve(lis); err != nil {
    log.Fatal(err)
  }
}

Instrument the client

Attach the client stats handler when you create the connection with grpc.NewClient. Pass the incoming request context into every RPC so the trace continues across the call.

package main

import (
  "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
  "google.golang.org/grpc"
  "google.golang.org/grpc/credentials/insecure"
)

func newClient(target string) (*grpc.ClientConn, error) {
  return grpc.NewClient(
    target,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
  )
}

Configure and run

Set the ingest environment variables for your deployment target, then start the service. Every tab is a complete path.

export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.osuite.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="x-osuite-ingest-token=<your-ingest-token>"
go run .

docker run \
  -e OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.osuite.io:443" \
  -e OTEL_EXPORTER_OTLP_HEADERS="x-osuite-ingest-token=<your-ingest-token>" \
  checkout-service

Create a secret with the ingest header, then reference it from the pod spec.

kubectl create secret generic osuite-ingest \
  --from-literal=otlp-headers="x-osuite-ingest-token=<your-ingest-token>"

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "https://ingest.<region>.osuite.io:443"
  - name: OTEL_EXPORTER_OTLP_HEADERS
    valueFrom:
      secretKeyRef:
        name: osuite-ingest
        key: otlp-headers

Framework notes

Use the stats handlers, not the interceptors. otelgrpc.UnaryServerInterceptor and otelgrpc.StreamServerInterceptor are deprecated. NewServerHandler and NewClientHandler are stats handlers that cover unary and streaming RPCs in one registration.

Create clients with grpc.NewClient. grpc.Dial and grpc.DialContext are deprecated. grpc.NewClient is the current constructor and takes grpc.WithStatsHandler the same way.

Context propagation is automatic. The stats handlers inject and extract W3C trace context in gRPC metadata, so a server span and its downstream client spans join the same trace with no manual metadata handling — provided both sides use the handlers and the caller passes the incoming context into outbound RPCs.

Verify in Osuite

What you should see

Call a method on the server. The service appears in APM within a minute, and the RPC shows as a span in the trace explorer with the full gRPC method name and status code. A client and server that both use the handlers appear as two spans in one trace.

Troubleshooting

No spans on the server. The stats handler was not passed to grpc.NewServer, or you are using the deprecated interceptors alongside a mismatched registration.
Client and server appear as separate traces. One side is missing its stats handler, or the client built the RPC from a background context instead of the incoming request context.

No data in Osuite after a few minutes? Work through these checks.

Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
Network — confirm the host has outbound access to the ingest endpoint on port 443.
Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.