Skip to content
Osuite Osuite Docs

Collect Google Cloud logs in Osuite

Overview

Section titled “Overview”

Google Cloud Logging aggregates logs from Compute Engine, GKE, Cloud Run, Cloud SQL, and every other Google Cloud service. You export those logs to a Pub/Sub topic with a logging sink, and the OpenTelemetry Collector’s googlecloudpubsub receiver pulls them from a subscription and forwards each record to Osuite through the otlp/osuite exporter.

Prerequisites

Section titled “Prerequisites”
  • An OpenTelemetry Collector from the Contrib distribution (otelcol-contrib) that includes the googlecloudpubsub receiver and the google_cloud_logentry_encoding extension.
  • Your Osuite ingest endpoint and token.
  • A Pub/Sub topic, a subscription, and a Cloud Logging sink that writes to the topic. Create them with gcloud:
Terminal window
gcloud pubsub topics create osuite-logs
gcloud pubsub subscriptions create osuite-logs-sub --topic osuite-logs
gcloud logging sinks create osuite-sink \
  pubsub.googleapis.com/projects/<project-id>/topics/osuite-logs \
  --log-filter='resource.type="gce_instance" OR resource.type="k8s_container"'
  • Grant the sink’s writer identity permission to publish. The create command prints a serviceAccount: address — grant it roles/pubsub.publisher on the topic:
Terminal window
gcloud pubsub topics add-iam-policy-binding osuite-logs \
  --member='serviceAccount:<sink-writer-identity>' \
  --role='roles/pubsub.publisher'
  • A service account the Collector authenticates as, granted roles/pubsub.subscriber on the subscription. Credentials reach the Collector through Application Default Credentials (ADC): a key file referenced by GOOGLE_APPLICATION_CREDENTIALS off GCP, or an attached identity (Workload Identity on GKE) on GCP.

Collector config

Section titled “Collector config”

This configuration pulls log entries from the subscription, decodes them with the LogEntry encoding extension, and forwards them to Osuite. Set project and subscription to your own.

extensions:
  google_cloud_logentry_encoding:


receivers:
  googlecloudpubsub:
    project: <project-id>
    subscription: projects/<project-id>/subscriptions/osuite-logs-sub
    encoding: google_cloud_logentry_encoding


processors:
  batch:


exporters:
  otlp/osuite:
    endpoint: ingest.<region>.osuite.io:443
    tls:
      insecure: false
    headers:
      x-osuite-ingest-token: <your-ingest-token>


service:
  extensions: [google_cloud_logentry_encoding]
  pipelines:
    logs:
      receivers: [googlecloudpubsub]
      processors: [batch]
      exporters: [otlp/osuite]

The subscription value must be the fully qualified name (projects/<project-id>/subscriptions/<name>). The google_cloud_logentry_encoding extension parses each Cloud Logging LogEntry into an OTLP log record, mapping severity, timestamp, and resource labels.

Verify in Osuite

Section titled “Verify in Osuite”

What you should see

Cloud Logging records appear in the Logs Explorer within a minute or two, carrying their Google Cloud severity and resource labels (project, zone, resource type). Filtering by cloud.provider = gcp isolates them.

Troubleshooting

Section titled “Troubleshooting”

No data usually means a permission or routing gap. Confirm the sink is actually writing to the topic (gcloud logging sinks describe osuite-sink), that the sink writer identity has roles/pubsub.publisher, and that the Collector’s service account has roles/pubsub.subscriber on the subscription. A decode error in the Collector logs means the encoding extension is not wired into service.extensions or the receiver’s encoding key. Confirm the subscription has undelivered messages with gcloud pubsub subscriptions pull.

No data in Osuite after a few minutes? Work through these checks.

  • Endpoint — confirm the exporter targets exactly ingest.<region>.osuite.io:443 for your region, over TLS.
  • Token — confirm the ingest header carries a valid <your-ingest-token> and has not been rotated.
  • Pipeline — confirm the signal you expect (traces, logs, or metrics) is wired into an active pipeline with the otlp/osuite exporter attached.
  • Export errors — check the application or Collector logs for OTLP export failures (auth, DNS, TLS, connection refused).
  • Network — confirm the host has outbound access to the ingest endpoint on port 443.
  • Timing — allow up to a minute for the first data to appear before assuming a failure.

Still stuck? Ask the Investigation Agent or contact support.

Next steps

Section titled “Next steps”