Data has become the new gold, powering many industries and even making them dependent on it. Tech companies develop products that collect and process information, using it to make smart decisions. On the other end, users expect real-time insights into how they interact with these products. They want all the relevant data in one place, instantly—without the clutter.

This creates several technical challenges for development teams, but I want to focus on one key issue: queries. When dealing with small amounts of data and simple filters, things are straightforward. But when you're working with hundreds of terabytes of data and need complex slice & dice capabilities with just a few minutes of delay, things get much trickier.

For example, advertisers using Taboola’s platform expect a single dashboard showing all their campaign data—both operational details (like name, creation date, and status) and performance metrics (such as clicks, conversions, and costs). Over time, as data volumes grew and queries became more complex, handling them efficiently became a major challenge—one that every data-driven company faces.

In this session, I'll share the solution we developed to handle large-scale queries quickly and efficiently.