Arnav Naresh Borkar
Data Ecosystem Lead, e6data
Actions
Arnav Borkar is a Performance & Research Engineer at e6data, where he contributes to the core development and optimisation of high-performance SQL query engine for large-scale analytics. He is a Computer Science graduate from BITS Pilani, where he nurtured his interest in distributed systems and data storage technologies through his work at DaSH Lab. Arnav has also published research on parallel file systems at IEEE Cluster, Santa Fe and presented part of this work at SuperComputing ’23 in Denver.
Links
Beyond Tables: Where Should Semantics Live in the Open Lakehouse?
Open table formats like Apache Iceberg, Apache Hudi, and Delta Lake have transformed modern data platforms by separating storage from compute and enabling multiple engines to operate on the same data. In this architecture, the catalog layer became the control plane for the open lakehouse—standardizing how engines discover tables and metadata.
However, lakehouses still primarily expose tables and SQL. While this works for analysts, newer consumers—applications, data agents, and AI-native systems—need semantic context such as entities, relationships, and shared metric definitions.
Today this logic is scattered across tools and pipelines. This raises a key question:
The catalog standardized how lakehouses discover tables. Should it also define what those tables mean?
In this talk, we explore architectural options for introducing semantics into the lakehouse:
- BI-centric semantics
- Pipeline-centric semantics
- Engine-centric semantics
- Catalog-centric semantics
- Service-based semantics
and discuss the trade-offs for governance, interoperability, and multi-engine environments.
Metadata at Scale: Tackling Iceberg Tables with Tens of Millions of Files
Modern data lakes promise near-infinite scalability, but in practice, metadata often becomes the silent bottleneck. At e6data, we ran headfirst into this challenge while working with a massive enterprise Iceberg table containing ~45 million active files, 50TB of data, and a staggering 5TB metadata log.
On the surface, this looked like a success story of scale, until queries started failing. A naïve scan would force the engine to load tens of millions of file-level statistics into memory, eventually exhausting even the largest executors. The result: a system that could store data at scale but struggled to query it efficiently.
In this session, we’ll unpack how we diagnosed and solved this metadata explosion problem by rethinking how Iceberg metadata should be read and pruned:
1. Why append-heavy ingestion and massive file counts amplify metadata growth.
2. How streaming metadata reads - rather than materialising everything in memory unlocked scalability.
3. The layered pruning approach we built to progressively narrow the search space without overwhelming the engine.
4. How treating metadata as first-class data allowed us to apply the same advanced pruning and caching strategies that power our query engine.
This isn’t just theory, it’s a story of lessons learned from production-scale Iceberg, where the challenge wasn’t the data itself but the metadata meant to manage it.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top