Manual OpenTelemetry Collector on Kubernetes

The osuite-k8s Helm chart is the supported path and handles the collector topology for you. If you cannot use Helm, or you run your own collector deployment, you can wire the same signals by hand with the OpenTelemetry Collector Contrib distribution.

Kubernetes monitoring needs two collector roles because the receivers have different scopes:

• A DaemonSet collector runs on every node and reads node-local data: the kubelet’s metrics and the pod log files on that node.
• A single-replica Deployment collector runs once per cluster and reads cluster-scoped data: object state and events.

Run both against the OpenTelemetry Collector Contrib image, which includes every receiver and processor used below.

Prerequisites

• A Kubernetes cluster and kubectl.
• A ServiceAccount bound to a ClusterRole that can get, list, and watch nodes, pods, namespaces, workloads, events, and nodes/stats and nodes/proxy.
• The KUBE_NODE_NAME environment variable injected into the DaemonSet pods via the downward API (spec.nodeName), so kubeletstats and k8sattributes can target the local node.

DaemonSet collector: node metrics and container logs

The kubeletstats receiver reads node, pod, and container metrics from the local kubelet. The filelog receiver tails pod log files, and k8sattributes enriches both with workload identity.

receivers:
  kubeletstats:
    collection_interval: 30s
    auth_type: serviceAccount
    endpoint: https://${env:KUBE_NODE_NAME}:10250
    insecure_skip_verify: true
    metric_groups: [node, pod, container, volume]
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    exclude:
      - /var/log/pods/*_otel-*/*/*.log
    start_at: end
    include_file_path: true
    operators:
      - type: container
        id: container-parser

processors:
  k8sattributes:
    auth_type: serviceAccount
    filter:
      node_from_env_var: KUBE_NODE_NAME
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
        - k8s.node.name
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: resource_attribute
            name: k8s.pod.name
          - from: resource_attribute
            name: k8s.namespace.name
  batch: {}

exporters:
  otlp/osuite:
    endpoint: ingest.<region>.osuite.io:443
    tls:
      insecure: false
    headers:
      x-osuite-ingest-token: <your-ingest-token>

service:
  pipelines:
    metrics:
      receivers: [kubeletstats]
      processors: [k8sattributes, batch]
      exporters: [otlp/osuite]
    logs:
      receivers: [filelog]
      processors: [k8sattributes, batch]
      exporters: [otlp/osuite]

Caution

The filelog exclude above drops the collector’s own log files so it does not ingest its own output. Adjust the glob to match the namespace and pod naming you deploy this collector under.

Deployment collector: cluster metrics and events

The k8s_cluster receiver emits cluster-level metrics (deployment, node, and pod state). The k8sobjects receiver watches events. Both are cluster-scoped, so this collector must run as a single replica to avoid duplicate data.

receivers:
  k8s_cluster:
    auth_type: serviceAccount
    collection_interval: 30s
    node_conditions_to_report: [Ready, MemoryPressure, DiskPressure]
    allocatable_types_to_report: [cpu, memory]
  k8sobjects:
    auth_type: serviceAccount
    objects:
      - name: events
        group: events.k8s.io
        mode: watch

processors:
  batch: {}

exporters:
  otlp/osuite:
    endpoint: ingest.<region>.osuite.io:443
    tls:
      insecure: false
    headers:
      x-osuite-ingest-token: <your-ingest-token>

service:
  pipelines:
    metrics:
      receivers: [k8s_cluster]
      processors: [batch]
      exporters: [otlp/osuite]
    logs:
      receivers: [k8sobjects]
      processors: [batch]
      exporters: [otlp/osuite]

For the enrichment and severity handling that the chart applies to events and container logs, see Kubernetes events and Container logs.

Verify in Osuite

What you should see

Node and pod metrics from kubeletstats, cluster-state metrics from k8s_cluster, and container logs and events appear in Osuite within a minute, each tagged with k8s.namespace.name and k8s.pod.name.

Troubleshooting

• kubeletstats returns nothing — confirm KUBE_NODE_NAME is set from spec.nodeName and the ServiceAccount can reach nodes/stats.
• Duplicate cluster metrics or events — run exactly one replica of the Deployment collector; k8s_cluster and k8sobjects are singletons.
• Logs missing workload identity — confirm the k8sattributes processor is in the logs pipeline and the pod’s ServiceAccount can list pods.

No data in Osuite after a few minutes? Work through these checks.

• Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
• Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
• Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
• Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
• Network — confirm the host has outbound access to the ingest endpoint on port 443.
• Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.