Skip to content
Osuite Osuite Docs

Log processing pipelines and retention

Overview

Section titled “Overview”

A log pipeline is everything that happens to a record between collection and storage. There are two stages: processing in the OpenTelemetry Collector, which runs before logs leave your infrastructure, and retention and archival in Osuite, which runs after logs are ingested.

Processing in the Collector

Section titled “Processing in the Collector”

Processors run in order between a pipeline’s receivers and its otlp/osuite exporter. Each one sees every record the receivers produce, so redaction and filtering here happen before any data reaches Osuite. Order matters: drop records first, then transform what remains, then batch.

Redact sensitive data

Section titled “Redact sensitive data”

Use the transform processor to remove or mask attributes with OTTL. This example deletes an authorization header attribute and masks bearer tokens anywhere in the attribute values:

processors:
  transform/redact:
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "authorization")
          - replace_all_patterns(attributes, "value", "(?i)bearer\\s+[a-z0-9._-]+", "bearer [REDACTED]")

The redaction processor is an alternative when you want allowlist-based masking (keep only named keys, block values matching a pattern). Redacting in the Collector keeps sensitive data out of Osuite entirely.

Filter unwanted logs

Section titled “Filter unwanted logs”

Use the filter processor to drop records you do not want to store — health checks, high-volume debug lines, or a noisy path. Records matching any condition are dropped:

processors:
  filter/drop_health:
    logs:
      log_record:
        - 'IsMatch(attributes["http.target"], "/health.*")'
        - 'severity_number < SEVERITY_NUMBER_INFO'

Enrich and normalize

Section titled “Enrich and normalize”

Use transform to set severity, rename fields, or add context, and resourcedetection to tag records with host and cloud metadata. Normalizing severity here makes level-based filtering in the explorer consistent across sources.

Wire it together

Section titled “Wire it together”

Processors take effect only when listed in the pipeline. Put filtering before transformation, and keep batch last:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317


processors:
  filter/drop_health:
    logs:
      log_record:
        - 'IsMatch(attributes["http.target"], "/health.*")'
  transform/redact:
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "authorization")
  batch:


exporters:
  otlp/osuite:
    endpoint: ingest.<region>.osuite.io:443
    tls:
      insecure: false
    headers:
      x-osuite-ingest-token: <your-ingest-token>


service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [filter/drop_health, transform/redact, batch]
      exporters: [otlp/osuite]

Retention and archival in Osuite

Section titled “Retention and archival in Osuite”

Retention and archival happen on the Osuite side, not the Collector — they apply to logs after ingestion.

  • Retention — ingested logs are queryable in the Logs Explorer for a retention window set for your workspace. Beyond that window, logs age out of the searchable store.
  • S3 archival — on enterprise plans, Osuite can sync logs to your own S3-compatible bucket for long-term or compliance storage beyond the retention window. This is configured by the Osuite team on request — contact us to set it up.

Next steps

Section titled “Next steps”