Celery OpenTelemetry instrumentation

Instrument Celery workers with OpenTelemetry so each task runs in its own trace, linked to the request that enqueued it. Celery’s default prefork pool forks worker processes after startup, so the SDK must be configured inside each worker rather than once at import time. This guide configures telemetry from the worker_process_init signal.

Fastest path. Run this in your AI IDE and the Instrumentation Agent wires this service up for you.

/osuite-instrument celery

The agent installs the Celery instrumentor and wires the worker_process_init signal for you.

Prefer to do it by hand? Follow the manual steps below — they are the complete path.

Prerequisites

• A Celery application with a broker configured (Redis, RabbitMQ, or similar).
• Python 3.8 or newer.
• Your Osuite ingest token and region.

Install and configure

1. Install the distro and OTLP exporter, then let bootstrap add the Celery instrumentor.

   pip install opentelemetry-distro opentelemetry-exporter-otlp
   opentelemetry-bootstrap -a install

2. Create telemetry.py and connect the SDK setup to worker_process_init. This runs once in each forked worker, after the fork.

   import logging
   
   from celery.signals import worker_process_init
   
   from opentelemetry import trace
   from opentelemetry.sdk.trace import TracerProvider
   from opentelemetry.sdk.trace.export import BatchSpanProcessor
   from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
   from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
   from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
   from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
   from opentelemetry.instrumentation.celery import CeleryInstrumentor
   
   
   @worker_process_init.connect(weak=False)
   def init_telemetry(*args, **kwargs):
       tracer_provider = TracerProvider()
       tracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
       trace.set_tracer_provider(tracer_provider)
   
       logger_provider = LoggerProvider()
       logger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))
       logging.getLogger().addHandler(LoggingHandler(logger_provider=logger_provider))
   
       CeleryInstrumentor().instrument()

3. Import telemetry from the module that defines your Celery app so the signal is registered before workers start.

   import telemetry  # noqa: F401
   
   from celery import Celery
   
   app = Celery("tasks", broker="redis://localhost:6379/0")

Run the worker

The worker_process_init handler configures telemetry, so start the worker with your normal command. Set the exporter environment variables in the worker’s environment.

Bare host

export OTEL_SERVICE_NAME="<service-name>"
export OTEL_RESOURCE_ATTRIBUTES="service.environment=production"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.osuite.io:443"
export OTEL_EXPORTER_OTLP_HEADERS="x-osuite-ingest-token=<your-ingest-token>"

celery -A tasks worker --loglevel=INFO

Docker

FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt opentelemetry-distro opentelemetry-exporter-otlp \
    && opentelemetry-bootstrap -a install
COPY . .

CMD ["celery", "-A", "tasks", "worker", "--loglevel=INFO"]

Pass the exporter variables at runtime:

docker run \
  -e OTEL_SERVICE_NAME="<service-name>" \
  -e OTEL_RESOURCE_ATTRIBUTES="service.environment=production" \
  -e OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.osuite.io:443" \
  -e OTEL_EXPORTER_OTLP_HEADERS="x-osuite-ingest-token=<your-ingest-token>" \
  myimage

Kubernetes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-worker
spec:
  template:
    spec:
      containers:
        - name: worker
          image: myimage
          command: ["celery", "-A", "tasks", "worker", "--loglevel=INFO"]
          env:
            - name: OTEL_SERVICE_NAME
              value: "<service-name>"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "service.environment=production"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "https://ingest.<region>.osuite.io:443"
            - name: OTEL_EXPORTER_OTLP_HEADERS
              value: "x-osuite-ingest-token=<your-ingest-token>"

Framework gotchas

Configure telemetry after the fork, not at import

The prefork pool forks worker processes from the parent. A TracerProvider and BatchSpanProcessor created at import time live in the parent and their background export thread does not survive the fork, so tasks emit no spans. Always create the providers inside the worker_process_init handler, which runs in each worker after it forks.

Trace context propagates from the producer

When the web service that enqueues a task is also instrumented, the Celery instrumentor propagates trace context through the broker. The task span becomes a child of the request that scheduled it, so the whole flow appears in one trace.

Other pool types

With the solo, threads, or gevent pools there is no fork, so you can instead configure the SDK once at startup. The worker_process_init approach works for every pool type and is the safe default.

Verify in Osuite

What you should see

Enqueue a task. A trace for the task appears in APM within a minute, showing the task name and any database or HTTP spans inside it. If the producer is instrumented, the task span is nested under the request that scheduled it. Task logs appear in the Logs explorer with the matching trace ID.

Troubleshooting

No data in Osuite after a few minutes? Work through these checks.

• Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
• Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
• Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
• Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
• Network — confirm the host has outbound access to the ingest endpoint on port 443.
• Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.

Next steps

Custom instrumentation Add manual spans and metrics inside your tasks.

Explore traces Open the trace explorer and service map in Osuite.

Correlate logs with traces Link application logs to the traces that produced them.