Skip to content
Osuite Osuite Docs

Collect AWS CloudWatch logs in Osuite

Overview

Section titled “Overview”

Amazon CloudWatch Logs is the default destination for logs from EC2, ECS, Lambda, RDS, API Gateway, and most other AWS services. The OpenTelemetry Collector reads those log groups and forwards each record to Osuite through the otlp/osuite exporter, where they correlate with the rest of your telemetry.

There are two Collector-based paths:

  • Pull with the awscloudwatch receiver — the Collector polls CloudWatch on a schedule. Simplest to run: no inbound endpoint, no extra AWS resources. CloudWatch charges per API request.
  • Push with Firehose — a CloudWatch subscription filter streams log events through Amazon Data Firehose to the Collector’s awsfirehose receiver. Lower latency and no polling cost, but the Collector must expose a public HTTPS endpoint that Firehose can reach.

Prerequisites

Section titled “Prerequisites”
  • An OpenTelemetry Collector from the Contrib distribution (otelcol-contrib) that includes the awscloudwatch and awsfirehose receivers.
  • Your Osuite ingest endpoint and token.
  • AWS credentials available to the Collector through the standard AWS SDK chain: environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), a shared credentials file, or an instance/role identity (IMDS on EC2, IRSA on EKS).
  • For the pull path, an IAM policy granting read access to CloudWatch Logs:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:FilterLogEvents",
        "logs:GetLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Pull with the awscloudwatch receiver

Section titled “Pull with the awscloudwatch receiver”

This configuration auto-discovers log groups by prefix, polls them on an interval, and forwards the records to Osuite. Set region to the AWS region whose logs you want.

receivers:
  awscloudwatch:
    region: us-east-1
    logs:
      poll_interval: 1m
      max_events_per_request: 1000
      groups:
        autodiscover:
          limit: 100
          prefix: /aws/eks/
          streams:
            prefixes: [kube-apiserver]


processors:
  batch:


exporters:
  otlp/osuite:
    endpoint: ingest.<region>.osuite.io:443
    tls:
      insecure: false
    headers:
      x-osuite-ingest-token: <your-ingest-token>


service:
  pipelines:
    logs:
      receivers: [awscloudwatch]
      processors: [batch]
      exporters: [otlp/osuite]

To collect from specific log groups instead of discovering by prefix, replace autodiscover with a named block that maps each log group name to an optional stream filter. Use named when you want a fixed, small set of groups; use autodiscover to follow every group under a prefix as new ones appear.

Lambda writes each function’s logs to a /aws/lambda/<function-name> group, so setting prefix: /aws/lambda/ collects logs from every Lambda function in the region.

Push with Firehose

Section titled “Push with Firehose”

For lower latency, stream CloudWatch Logs through Amazon Data Firehose to the awsfirehose receiver. The receiver decodes the Firehose payload with the awslogs_encoding extension set to format: cloudwatch.

extensions:
  awslogs_encoding/cloudwatch:
    format: cloudwatch


receivers:
  awsfirehose:
    endpoint: 0.0.0.0:4433
    encoding: awslogs_encoding/cloudwatch
    access_key: <your-firehose-access-key>
    tls:
      cert_file: /etc/otel/server.crt
      key_file: /etc/otel/server.key


processors:
  batch:


exporters:
  otlp/osuite:
    endpoint: ingest.<region>.osuite.io:443
    tls:
      insecure: false
    headers:
      x-osuite-ingest-token: <your-ingest-token>


service:
  extensions: [awslogs_encoding/cloudwatch]
  pipelines:
    logs:
      receivers: [awsfirehose]
      processors: [batch]
      exporters: [otlp/osuite]

On the AWS side, create a Firehose delivery stream with an HTTP endpoint destination pointing at the Collector’s public HTTPS address, set its access key to match access_key, then add a CloudWatch Logs subscription filter that delivers the target log groups to that stream. The awslogs_encoding extension is alpha — pin the Collector version.

S3-stored logs

Section titled “S3-stored logs”

The awss3 receiver exists but is a replay receiver: it reads back telemetry the Collector itself previously exported to S3 in OTLP-compatible encodings, not arbitrary log objects written by other AWS services (ELB access logs, CloudTrail, VPC flow logs). For those, the practical path is the same Firehose route above, or a Lambda that reads the object and posts to the awsfirehose endpoint. There is no turnkey receiver that tails a general S3 log bucket.

Verify in Osuite

Section titled “Verify in Osuite”

What you should see

CloudWatch log records appear in the Logs Explorer within a minute or two, tagged with their originating log group and stream. Filtering by cloud.provider = aws isolates them from the rest of your logs.

Troubleshooting

Section titled “Troubleshooting”

Empty results on the pull path usually mean the IAM policy is missing logs:FilterLogEvents, or the region/prefix does not match any log group. On the push path, confirm Firehose reaches the Collector’s public endpoint, the access_key values match, and the subscription filter is attached to the right log groups. Watch the Collector logs for AccessDenied or ThrottlingException and widen poll_interval if you hit CloudWatch API limits.

No data in Osuite after a few minutes? Work through these checks.

  • Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
  • Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
  • Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
  • Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
  • Network — confirm the host has outbound access to the ingest endpoint on port 443.
  • Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.

Next steps

Section titled “Next steps”