Thanos at Scale: Query Optimization in Global Kubernetes Environments

Consider a setup where Thanos is deployed in Kubernetes clusters globally. A significant challenge arises when queries made on the global query layer need to be fanned out to all these clusters. This process, spreading queries widely regardless of where the data actually is, turns out to be costly and inefficient.

In our proposed talk, we'll discuss eBay's approach to this issue. Our Thanos deployment generates indexes when new time series are discovered, which are then sent to a centralized server. Utilizing these indexes, we've developed a specialized query layer. This layer effectively determines which clusters contain the needed data, thereby directing queries only to those relevant clusters. This strategy significantly reduces unnecessary global query distribution, leading to more efficient and faster data retrieval. We plan to explore the architecture of this solution and its impact at eBay, providing insights on how we manage Thanos in a large-scale, distributed setup.