Federated search over distributed observability data

While operating Observability at scale and in huge volumes, it is optimal to keep the data distributed at edge, in-cluster, intermediate storage, separate storage for structure and unstructured logs, tracing and metrics. This drives the need for federated query and search over distributed data stores, aggregate and stitch data and perform analytics. This talk explores the use of technologies such as OpenTelemetry, Apache Arrow and Apache Presto to perform the federated search and how it can be used to improve speed, performance and significantly bring down Observability costs (for compute, networking and storage) and illustrates the same using a case study from Intuit.

Intuit journey to unified observability at scale: Challenges, benefits and lessons learned

At Intuit we have ~320 Kubernetes clusters running with ~8000 services and ~40 addons in a cluster which generate ~2 billion active time series metrics, 10 million Trace spans/sec and ~ 1.2PB of log data ingested (peak) in a single day. This talk focuses on Intuit’s journey from standalone, siloed, proprietary solutions for logs, metrics and traces to a unified observability solution. This is made possible with a data management architecture that enables seamless navigation and correlation between different observability pillars, usage of AI/ML techniques to quickly detect and isolate problems, UX that brings all the elements of data discovery with an interactive experience and high level features like golden signals, RUM (real user monitoring) and FCI (failed customer interactions). All of this leading to significantly lower MTTD and MTTI. We also discuss the challenges, choices, trade offs, benefits and lessons learned during this journey.

Benefits

This talk is relevant to infrastructure engineering teams, observability platform developers/operators/architects and end users of observability platforms, especially teams looking to move to OpenTelemetry and evaluating different open source projects.

Observability has become a key to engineering effectiveness and operational excellence. Effective observability relies on comprehensive coverage of telemetry data in terms of collection, storage and analysis of raw data and the need to correlate all of these and navigate between them seamlessly in an intuitive UI.

This talk is going to present a high level approach and architecture for unified observability, the data processing pipelines, and data storage architectures based on CNCF technologies like OpenTelemetry, Fluentd, Kubernetes, Argo and open source technologies like Apache Iceberg, Parquet, Datafusion, Apache Arrow, Mimir and Grafana with challenges, trade offs, choices and lessons learned so far in our journey. The focus will be on the different capabilities that power the unified observability platform at intuit.

This talk is also going to focus on the unified experience powered by open source technologies that brings together various techniques that enable developers to correlate different data sets to quickly identify and isolate the problem.