5 Key Observability Trends for 2022

In the coming year, organizations will seek to simplify, optimize, and consolidate observability through a mix of new tools and practices.

I’ve argued in the past that observability is, at its core, a data analytics problem. The formal definition of observability tends to center on the external outputs of IT systems. I use a slightly broader definition of observability: “The capability to allow a human to ask and answer questions about the system”. I like this definition because it suggests that observability should be incorporated as part of the system design (rather than being bolted on as an afterthought) and because it underscores the need for engineers and system administrators to bring an analytics mindset to the challenge of enabling observability.

In the coming year, look for organizations to deepen and diversify their telemetry data usage, while consolidating their tooling, in an attempt to level up their observability. Technologies such as eBPF and OpenTelemetry will lower the barrier to entry on instrumentation, and matured data analytics practices will enable IT and DevOps teams to identify and respond to issues more quickly and effectively.

Broader adoption of distributed tracing

Distributed tracing can open up a whole new world of observability into numerous processes beyond IT monitoring, in areas as diverse as developer experience, business, and FinOps. Distributed tracing relies on instrumenting application with the mechanics of propagating context when executing requests. You can easily use the context propagation mechanism for many other processes, such as tracking resource attribution or capacity planning information per product line or per customer account.

Data privacy compliance is another extremely useful application of distributed tracing. In light of emerging compliance regulations such as GDPR and CCPA, data privacy is a huge priority, and this challenge is exacerbated by the fact that low-level storage is often unaware of user context. By propagating user IDs from downstream tiers to data storage tiers, distributed tracing can help organizations to better enforce their data privacy policies.

Movement beyond the ‘three pillars’ of observability

These three pillars continue to be critically important. But it’s important not to be confined by the “three pillars” paradigm and to choose the right telemetry data for your needs. In the coming year, I expect we’ll be seeing more organizations embrace additional types of observability signals, including events, snapshots and continuous profiling.

It is also important to remember that the “three pillars,” or any other telemetry signal for that matter, is just the raw data. As I wrote above, I firmly believe that observability is a data analytics problem, and as such, it is about proactively extracting insights out of that raw data, similar to BI analysts in a way. In December, I interviewed Frederic Branczyk, the founder of Polar Signals and a passionate advocate for observability. He shared the gap he sees in companies today:

We pretend in our observability bubble that everybody has well-instrumented applications with distributed tracing and structured logging. But the reality is, when I look at a typical startup, they may not even be monitoring at all. They’re waiting for their customers to tell them something is wrong before they start investigating.

More momentum behind eBPF

Because eBPF works across different types of traffic, it helps organizations to meet their goal of unified observability. For instance, DevOps engineers might use eBPF to collect full body trace requests, database queries, HTTP requests, or gRPC streams. They can also use eBPF to collect resource utilization metrics, including CPU usage or bytes sent, allowing the organization to calculate relevant statistics and profile their data to understand the resource consumption of various functions. Additionally, eBPF can handle encrypted traffic.

Netflix recently published a blog about how the company is using eBPF to capture network insights. According to the company, the use of eBPF has been highly efficient, consuming less than one percent of CPU and memory in any instance.

Unification of siloed tools

We saw this unification trend in the past year, with major vendors such as Grafana Labs, Datadog, AppDynamics, and my company Logz.io coming out of their respective specialty domains in log analytics, infrastructure monitoring, APM, or others, and expanding into a more comprehensive observability offering. We’ll see this trend accelerating in 2022, adapting to the changing observability needs and changing the competitive landscape.

Continued adoption of open source tools and standards

At the moment, the open source landscape for observability is quite dynamic, with a number of important projects emerging in just the past couple of years. It can sometimes be difficult for DevOps and system administrators to keep these solutions straight (especially because many have adopted the naming convention of “OpenSomething”), but they are beginning to converge. Each day in 2022, we will move closer to something resembling open source standardization — and closer to the ideal of unified observability.

This article was originally published at InfoWorld.com, reprinted with permission from © IDG Communications, Inc.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store