Debug or File Export for internal metrics - service.telemetry.readers.periodic.exporter

[Michaelh12304-work]

Using otel contrib v.0.139.0

For pushing the internal collector metrics a configuration like this can be used

service:
  telemetry:
    metrics:  # Internal metrics for the collector 
      # Detailed for Dynatrace's dashboard
      # https://docs.dynatrace.com/docs/ingest-from/opentelemetry/collector/self-monitoring
      level: detailed 
      readers:
      # This is for pushing the internal metrics of the collector to Dynatrace.
        - periodic:
            interval: 5m 
            exporter:
              otlp:
                protocol: http/protobuf
                temporality_preference: delta
                endpoint: "https://XXXXXXXXXXXXXXXXXXXXXXXXXXX/v1/metrics" # Put your Dynatrace metrics endpoint here
                headers:
                  - name: Authorization
                    value: "Api-Token " # Put your Dynatrace token here

(1) I take it there is no way to reuse an exporter defined elsewhere in the file? It's a bit odd having to define it twice.
(2) How do I use a Debug or File exporter to verify what is sent? This seems to crash the collector even with a very basic setup

See doc : https://opentelemetry.io/docs/collector/internal-telemetry/

Thanks

[dmathieu]

There is a bit of a quirk here, which is that the internal telemetry uses the Go SDK, which has its own exporter interface.
So when you define an exporter for internal telemetry, you're configuring the Go SDK exporter.

That means you can use any exporter provided by the SDK, and configuration from the otel-configuration schema. Any collector-specific exporter cannot be used here.

If you look at the configuration schema, you will notice it does include an otlp_file/development exporter configuration.
https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema/tracer_provider.yaml#L425

However, that exporter is not yet implemented in otelconf (the Go SDK package that handles otel-configuration). So it cannot be used yet.
open-telemetry/opentelemetry-go#5408

Regarding the crashes you mention, those seem a bit harsh. Could you paste a stacktrace?

[Cloud-Architect-Emma]

To add to @dmathieu's excellent explanation, as a practical workaround while
otlp_file/development is not yet implemented, you can verify what internal
metrics are being sent by using a local OTLP receiver:

1. Run a second collector locally (or in the same config) that just logs everything:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4319

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [debug]

2. Then point your service.telemetry.metrics.readers at localhost:4319
   instead of Dynatrace temporarily.

This lets you inspect the exact payload being sent without needing the
otlp_file exporter. It's the standard debugging approach until
open-telemetry/opentelemetry-go#5408
is resolved.

Regarding reusing exporters defined elsewhere in the config, you're correct
that this is not currently possible. The internal telemetry pipeline uses the
Go SDK directly and has no access to the collector's exporter registry. It's
a known limitation and a reasonable feature request to file.

[Michaelh12304-work]

Very creative!

This is perhaps not the place for this, but my need was for this issue.

The key issue for my Dynatrace issue with metrics was one to use their very awkwardly named API which is something like /api/v2/metrics/v1 or something weird like that AND you need the advanced ingestion setting in the Dynatrace settings to capture the resource attributes, in fact to capture much of any attribute when using OTEL. This is going to be on by default going forward.

[Cloud-Architect-Emma]

Glad it helped! For the Dynatrace-specific issue you mentioned,
yes, the /api/v2/otlp endpoint (not /api/v2/metrics/ingest)
is the correct one for OTLP ingestion, and enabling the
"OpenTelemetry resource attributes" setting in Dynatrace is
indeed required to capture resource attributes properly.

Good catch for others who may hit the same issue!