Modern data lakes promise near-infinite scalability, but in practice, metadata often becomes the silent bottleneck. At e6data, we ran headfirst into this challenge while working with a massive enterprise Iceberg table containing ~45 million active files, 50TB of data, and a staggering 5TB metadata log.
On the surface, this looked like a success story of scale, until queries started failing. A naïve scan would force the engine to load tens of millions of file-level statistics into memory, eventually exhausting even the largest executors. The result: a system that could store data at scale but struggled to query it efficiently.

In this session, we’ll unpack how we diagnosed and solved this metadata explosion problem by rethinking how Iceberg metadata should be read and pruned:
1. Why append-heavy ingestion and massive file counts amplify metadata growth.
2. How streaming metadata reads - rather than materialising everything in memory unlocked scalability.
3. The layered pruning approach we built to progressively narrow the search space without overwhelming the engine.
4. How treating metadata as first-class data allowed us to apply the same advanced pruning and caching strategies that power our query engine.

This isn’t just theory, it’s a story of lessons learned from production-scale Iceberg, where the challenge wasn’t the data itself but the metadata meant to manage it.