[loadbalancingexporter] Logs - duplicate traffic when using routing_key: attributes

[nickinkaty]

Component(s)

exporter/loadbalancing

What happened?

Description

Duplicate log records are observed after recovering from a network outage while using persistent storage (file_storage) with loadbalancer exporter. The setup consists of Agent Collectors sending logs to Gateway Collectors, which use the loadbalancingexporter to forward traffic to backend collectors.

During testing, a network outage was simulated using a Kubernetes NetworkPolicy. After connectivity was restored, the Agent Collector appeared to resend previously sent logs, resulting in duplicate records.

Current gateway configuration:

routing_key: attributes
routing_attributes:
  - service.name
  - k8s.pod.name
  - log.body

Steps to Reproduce

Deploy Agent Collectors that send logs to Gateway Collectors.
Configure the Gateway Collectors with the loadbalancingexporter.
Generate log traffic.
Simulate a network outage using a Kubernetes NetworkPolicy.
Restore connectivity.
Observe duplicate logs at the backend.

Example duplicate log:

Date,Host,Service,Pod Name,Content
"2026-06-14T17:50:06.474Z","[REDACTED_HOST]","opentelemetry-collector-contrib","otel-gateway-sts-collector-1","Sun Jun 14 17:43:41 UTC 2026 counter=12494 node=[REDACTED_NODE] logtag=F log.file.path=[REDACTED_PATH] time=2026-06-14T17:43:41.358885254Z log.iostream=stdout gw.timestamp=2026-06-14T17:43:41Z"
"2026-06-14T17:49:55.692Z","[REDACTED_HOST]","opentelemetry-collector-contrib","otel-gateway-sts-collector-1","Sun Jun 14 17:43:41 UTC 2026 counter=12494 node=[REDACTED_NODE] logtag=F log.file.path=[REDACTED_PATH] time=2026-06-14T17:43:41.358885254Z log.iostream=stdout gw.timestamp=2026-06-14T17:43:41Z"
"2026-06-14T17:49:50.560Z","[REDACTED_HOST]","opentelemetry-collector-contrib","otel-gateway-sts-collector-1","Sun Jun 14 17:43:41 UTC 2026 counter=12494 node=[REDACTED_NODE] logtag=F log.file.path=[REDACTED_PATH] time=2026-06-14T17:43:41.358885254Z log.iostream=stdout gw.timestamp=2026-06-14T17:43:41Z"
"2026-06-14T17:49:45.446Z","[REDACTED_HOST]","opentelemetry-collector-contrib","otel-gateway-sts-collector-1","Sun Jun 14 17:43:41 UTC 2026 counter=12494 node=[REDACTED_NODE] logtag=F log.file.path=[REDACTED_PATH] time=2026-06-14T17:43:41.358885254Z log.iostream=stdout gw.timestamp=2026-06-14T17:43:41Z"

The same log is received multiple times several seconds apart.

Expected Result

After connectivity is restored, buffered logs should be sent once and routed consistently to the same backend collector.

Actual Result

After the network outage ends, the Agent Collector resends previously sent logs. The duplicate logs are routed to the same backend collector, indicating that routing is working correctly, but duplicate data is being retransmitted.

Collector version

0.154.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

load_balancing/gateway:
        protocol:
          otlp:
            sending_queue:
              enabled: false
            timeout: 3s
            tls:
              insecure: true
        resolver:
          k8s:
            ports:
              - 4317
            service: otel-gateway-sts-collector-headless.otel-system
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_elapsed_time: 0
          max_interval: 30s
        routing_attributes:
          - service.name
          - k8s.pod.name
          - log.body
        routing_key: attributes
        sending_queue:
          batch:
            flush_timeout: 3s
            max_size: 2500
            min_size: 1000
          num_consumers: 35
          queue_size: 4000000
          sizer: items
          storage: file_storage/telemetry-retention

    service:
      extensions:
        - file_storage/fingerprint
        - file_storage/telemetry-retention
      pipelines:
        logs/pod:
          exporters:
            - load_balancing/gateway
          processors:
            - memory_limiter
            - resource/default
            - resource/podlogs
            - resource/lb
            - k8sattributes
            - transform/redact
            - redaction/keys
          receivers:
            - filelog/podLogs

Log output

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @rlankfo

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

Thanks for the detailed report. A couple of observations that may help triage this:

• The duplicate examples appear roughly one retry interval apart, and the issue says they still route to the same backend collector. That makes routing_key: attributes itself look stable; the more likely area is retry / queue replay after an uncertain or partial export result.
• This also looks related to loadbalancingexporter: failed endpoint retries re-route already-delivered records, causing duplicates on healthy endpoints #47600, although this report is on v0.154.0, uses routing_key: attributes, and has persistent storage on the loadbalancing exporter queue.

Could you clarify where the simulated NetworkPolicy outage is applied: Agent -> Gateway, or Gateway -> backend collectors? It would also help to know whether the duplicates are visible at Gateway ingress, or only after the loadbalancingexporter sends to the backend.

For the next reproduction, useful telemetry/logs would be:

• Gateway collector logs around the outage/recovery, especially Exporting failed, Rejecting data, sending queue is full, or file storage/queue replay messages for load_balancing/gateway.
• The relevant exporter queue metrics before, during, and after recovery, especially accepted/sent/failed log records and queue size/capacity for load_balancing/gateway and its OTLP sub-exporters.
• Whether the behavior changes if the loadbalancing exporter sending_queue.storage is removed temporarily, or if retry_on_failure.max_elapsed_time is bounded for the test.

Given the current evidence, I would not classify this as a routing inconsistency yet. It looks more like at-least-once retry semantics surfacing as duplicate delivery after recovery, potentially in the same family as #47600.

[nickinkaty]

@JaredTan95 Thanks for the quick response. It’s definitely the retry_on_failure mechanism. Do you know if there’s a configuration change or fix I could make?