Collect Kafka logs with OpenTelemetry

Overview

Apache Kafka brokers write server logs through log4j, with server.log as the main file alongside controller.log and state-change.log. The OpenTelemetry Collector’s filelog receiver tails these files, parses the log4j lines, folds Java stack traces into single records, and ships them to Osuite over OTLP. This page collects logs; for broker, topic, and consumer-group metrics see Monitor Kafka with OpenTelemetry.

Prerequisites

Kafka writing broker logs to disk. The directory is set by LOG_DIR (or kafka.logs.dir); common paths are /opt/kafka/logs/, $KAFKA_HOME/logs/, and /var/log/kafka/. This is the operational log directory, not the topic data directory (log.dirs).
The OpenTelemetry Collector installed on the broker host, with read access to the log directory.
An Osuite ingest token and endpoint — see Getting Started.

Collector config

This otel-collector.yaml tails the broker log. The multiline block treats any line that does not start with a [timestamp] as a continuation, so a Java exception and its stack trace arrive as one record. The regex assumes Kafka’s default log4j pattern.

receivers:
  filelog/kafka:
    include:
      - /opt/kafka/logs/server.log
    start_at: end
    multiline:
      line_start_pattern: '^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\]'
    operators:
      - type: regex_parser
        regex: '^\[(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\] (?P<level>\w+) (?P<message>(?s).*?)(?: \((?P<logger>[^)]+)\))?$'
        timestamp:
          parse_from: attributes.timestamp
          layout_type: strptime
          layout: '%Y-%m-%d %H:%M:%S,%f'
        severity:
          parse_from: attributes.level

processors:
  batch: {}
  resourcedetection:
    detectors: [system]
  resource:
    attributes:
      - key: service.name
        value: kafka
        action: upsert

exporters:
  otlp/osuite:
    endpoint: ingest.<region>.osuite.io:443
    tls:
      insecure: false
    headers:
      x-osuite-ingest-token: <your-ingest-token>

service:
  pipelines:
    logs:
      receivers: [filelog/kafka]
      processors: [batch, resourcedetection, resource]
      exporters: [otlp/osuite]

Parsing and enrichment

Format assumption — the regex parses Kafka’s default log4j pattern ([%d] %p %m (%c)%n), for example [2025-10-15 12:34:56,789] INFO Starting controller (kafka.controller.KafkaController). If your log4j.properties uses a custom ConversionPattern, adjust the capture groups.
Multiline and stack traces — a Java exception spans many lines, none of which start with a timestamp. The line_start_pattern marks a new record only at a [timestamp] line, so the exception header and its at ... frames stay in one record’s body.
Severity — log4j level words (TRACE, DEBUG, INFO, WARN, ERROR, FATAL) map directly to the record’s SeverityNumber without an explicit mapping. See severity.
Timestamps — Kafka’s default pattern has no zone; the parser reads the time as the Collector host’s local time. Keep the Collector and broker in the same zone.
Other log files — to collect controller.log and state-change.log as well, add their paths to include; they share the same log4j format.
Host and service attributes — the resourcedetection and resource processors tag each record with the host name and a stable service.name.

Verify in Osuite

What you should see

Within a minute, Kafka broker records appear in the Logs Explorer tagged with service.name=kafka and the host name. Filtering by severity isolates ERROR entries, and a Java stack trace appears as a single multi-line record rather than one record per frame.

Troubleshooting

If each stack-trace line arrives as its own record, the line_start_pattern does not match your timestamp format — align it with the actual [timestamp] prefix. If the whole line stays in the body with no attributes, your ConversionPattern differs from the assumed default; test the regex against a real log line and adjust.

No data in Osuite after a few minutes? Work through these checks.

Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
Network — confirm the host has outbound access to the ingest endpoint on port 443.
Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.