Performance comparison OTLP over gRPC vs. HTTP

[breedx-splk]

Looking at the collector performance page https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/performance.md got me wondering if there's a measurable difference between OTLP/gRPC vs. OTLP/HTTP. Has anybody looked into that?

The java agent is likely switching to a low-dependency OTLP/HTTP exporter as default (gRPC pulls in a lot of deps!)...and I wonder if this matters through a performance lens.

[bogdandrutu]

In java even HTTP client brings lots of deps :) I think performance wise they are pretty similar.

[breedx-splk]

The number of dependencies has no direct impact on the performance tho, it's really more about the underlying protocols, encodings, and how its all put together.

In any case, it would be good to have numbers backing this up, right?

[ishankhare07]

Has anybody here any idea about the binary size implications in case of http vs grpc protocols – talking here in perspective of the go SDK

[utezduyar]

It would also be nice to see the network bandwith comparison. I would imagine it would make a difference if a commercial collector is used which charges their customer per CPU / network traffic.

[cforce]

any news here?

[julealgon]

@breedx-splk do you have any updates on this?

The question was asked way back in 2021 and I don't see any relevant responses here.

I have the exact same question, as the .NET stack is also changing its defaults from gRPC to http/protobuf in compliance with the new recommendation from the OTEL spec.

We have many projects today pushing telemetry via gRPC, but some were broken when the .NET Framework version of the OTLP exporter dropped support for gRPC: this left us with a mix of gRPC and http, which makes global configuration cumbersome.

I would like to default everything to use http/protobuf but I'm worried about the potential performance impact in the collector instance, since we already work in a fairly constrained environment resource-wise.

Our non-production environment is not super representative today of PROD, which makes this update even more dangerous for us: we will only really see the actual impact when we push to PROD.

@bogdandrutu do you have any thoughts on this matter, or know another contributor who could assist?

[breedx-splk]

Hey @julealgon. I really don't have an update on this. I can say that in the 4+ years since I started this discussion I haven't really heard any real-world users complaining or bragging about one proto over the other regarding performance or overhead. I probably have some blind spots, because I'm not an SRE and I focus more on language and client instrumentation than the collector.

While I think it would be nice to have a real benchmark from the community itself, it does look like if you scratch around on the web you can find some writing on this topic from various vendors. At the end of the day, the choice is super complicated and there isn't a single one-size-fits-all for this stuff. It's good that you've identified the environmental factors that are different between prod and non-prod...but as you know, that only makes your situation harder.

I understand being cautious, but hopefully your worry is founded on some experimental or other data and not just the cloudy rhetoric about gRPC being more efficient (not completely true!). The painful truth is this: If you're seriously resource constrained then ANY change could cause serious or negative impact. When possible, the move is often to give yourself a bunch more headroom (ram, cpu, instances, whatever) and then try the change, then establish a new baseline before tuning the resource allocation back down.