OpenTelemetry Roadmap and Latest Updates
TL;DR Key updates:
- OpenTelemetry has reached RC for Metrics
- Logs specification is stable, Logs Beta plans
- Adding Real User Monitoring support to OpenTelemetry
- Adding Continuous Profiling to OpenTelemetry
- Ease of Operation with Remote Agent Management
- Port 4317 is registered for OpenTelemetry
- eBPF and more updates
OpenTelemetry is one of the most fascinating and ambitious open source projects of this era. It’s currently the second most active project in the CNCF (the Cloud Native Computing Foundation), with only Kubernetes being more active.
I was at KubeCon Europe last month, delivering a talk on OpenTelemetry and it was amazing to see the full house and the excitement and interest around the project. At KubeCon I also got to meet the project maintainers and community, and I brought fresh updates that I’d like to share with you in this post, as well as updates from OpenTelemetry Community Day held this month.
I also devoted this month’s episode of OpenObservability Talks to OpenTelemetry project, where I hosted Alolita Sharma, co-chair of the CNCF Technical Advisory Group for Observability and a member of the OpenTelemetry Governance Committee, among her many roles. Some interesting insights came out of my discussion with Alolita, which I’ll also share in this post.
If you are new to OpenTelemetry, check out my beginner’s guide here.
OpenTelemetry has reached RC for Metrics
The biggest news out of KubeCon Europe 2022 in this domain was that OpenTelemetry Metrics is available in Release Candidate.
With that, the API and SDK specifications are stable and implemented in Java, .NET and Python.
The OpenTelemetry protocol, OTLP, is also stable and you can relay metrics data over it alongside traces and logs. The OpenTelemetry Collector supports metric pipelines.
There is full Prometheus support, both in the SDKs that can export metrics in Prometheus (OpenMetrics) format, and in the Collector which can receive and export metrics in Prometheus format.
You can read the announcement details here.
Logs specification is stable in OpenTelemetry’s protocol OTLP
Another major news is that the OTLP Logs Protocol has just turned Stable aligning with the data model.
When can we expect Logging to reach maturity? As Alolita shared on the podcast, recent discussions in the OpenTelemetry Governance Committee target Beta realistically by the end of 2022:
Realistically at this point in time, I don’t expect logging to be stable until the end of the year at the earliest but I’d say early next year.
More broadly, the logging efforts initially focused on integration with existing logging sources, which are still prevalent and are typically unstructured text-based flat files, to be able to fetch these logs and transmit over OTLP.
Now more focus is being put in the longer term vision of OpenTelemetry: to build a new strongly-typed and machine-readable logging storage and transmission format. Some work has begun on defining the semantic conventions for this logging format, and there’s a promising collaboration with Elastic Common Schema community to merge the two open standards.
You can hear more about the logging signal updates in the latest podcast episode.
Adding Real User Monitoring support to OpenTelemetry
Most of the efforts within OpenTelemetry have so far been primarily focused on server side monitoring. However, it is equally important to monitor client side interactions, especially if OpenTelemetry is to support Real User Monitoring (RUM) use cases. Just think of native mobile or web applications, which app owners typically monitor to understand the user interactions in the app, to count and segmenting new and returning users, or to identify pages with high error or latency rates.
This realization has led to a special interest group (SIG) that was founded in late 2021 and got active during 2022. While it’s somewhat possible to instrument client applications using OpenTelemetry’s current JavaScript instrumentation support, there are special concerns for RUM, such as propagating identifiers for the session, page or view, that need modeling as part of the OpenTelemetry specification. Alolita even proposed that RUM be modeled as a brand new signal type.
You can read more about the purpose of the SIG here, and are welcome to join the discussion on the #otel-client-side-telemetry channel in the CNCF slack.
Adding Continuous Profiling to OpenTelemetry
One of the key trends in observability is moving beyond the ‘three pillars’. As Tracing, Metrics and Logs signals are progressing nicely, the OpenTelemetry community is starting to look at the next telemetry signal to incorporate in the observability stack: Continuous Profiling. This was a hot topic during KubeCon Europe 2022, and in June 2022 a new working group was established for the new signal. The working group is still in its early days, establishing the goals, evaluating existing open and custom formats, and seeing how profiling should be modeled in the OpenTelemetry architecture. You are welcome to take part and influence the discussion: you can join the group calls and read the summary of past calls in this running document. You can also join the #otel-profiles channel on the CNCF slack.
Ease of Operation with Remote Agent Management
As the project matures and is put to use in more production environments, more focus is put on making it easy to operate in production. A Kubernetes operator was released, which helped ease the deployment. Alongside deployment ease, focus is now put on post-deployment considerations around remote agent configuration and management. A new working group was established to address that: Open Agent Management Protocol (OpAMP). You can find more details on GitHub and on the #otel-agentmanwg channel on the CNCF slack.
Port 4317 is registered for OpenTelemetry
Finally Port 4317 is now officially registered for OpenTelemetry, as approved by IANA. This is an important step in the standardization of OpenTelemetry.
More updates
These are the prominent updates, but certainly not an exhaustive list. For example, I strongly believe in the potential of eBPF for observability, and I was glad to see the growing interest in the community around this topic. It came up at KubeCon Europe, and was the topic of a dedicated breakout session at OTEL Community Day.
And that’s an important update in its own right: June 20th the OpenTelemetry community held its first OTEL Community Day, which took place in Austin, Texas, with very good workshops and discussions. Here’s a good summary with more updates from that day.
Want to learn more? Check out the OpenObservability Talks latest episode: OpenTelemetry and the Vision for Unified Open Observability on Apple, Spotify or other apps.
