Zero Rows, One Object — Both Systems Are Telling the Truth

There is a Parquet file in your Lakehouse right now holding data you told someone you deleted.

Not because a job failed. Run the query and you get zero rows; every engine agrees the row is gone. List the object versions in the bucket underneath and the file is still there, intact, readable by anyone with object access and by any engine pinned to an older snapshot. Both systems are working exactly as designed. Iceberg promised your readers would not see that row, and it kept the promise the instant the delete committed. It never said anything about bytes. The regulation is entirely about bytes.

What makes this hard is that the gap has no owner. There is no exception to catch, no failed job to page anyone about, and no dashboard that turns red. Most teams alarm on whether the VACUUM job succeeded, which tells you the job ran. It does not tell you the data is gone, and the table metadata will cheerfully report every row as deleted while the objects sit in storage. If your erasure evidence comes from the table, you are auditing the wrong system. The only honest answer comes from the bucket, and almost nobody asks it.

So we asked it. Two of us built harnesses that drive real Apache Iceberg tables through erasure requests in virtual time, then merged them, and measured how long a deleted row's bytes actually survive under different maintenance calendars and bucket settings. Eighty-two runs. Persistence checked by listing object versions, never by trusting metadata. Under documented defaults and the retention most deployments actually set, the bytes outlived the delete by 34 days against a 30-day window.

Then the result that changed how we think about this. We expected compaction to dominate, because compaction is what the entire ecosystem optimises: merge-on-read, deletion vectors, better clustering, smarter schedulers. Compaction turned out to be four of the thirty-four days. Everything else is waiting on snapshot expiry, on an orphan-cleanup job somebody deferred, and finally on a bucket lifecycle rule that lives in a different system under a different team's control. The work being poured into the first stage is competing for a quarter of a day.

We tested this the boring way, one variable at a time on the same table. Change the delete mode and the total moves by a day. Deletion vectors are genuinely worth adopting, and they rewrite far fewer bytes, but they will not move a compliance deadline and this talk will show you why the reason is structural rather than tunable. Change the data layout and the total does not move at all. Change one setting in the bucket and it runs from about a month to never finishing. That last sentence is not a figure of speech. There are two configurations where physical erasure cannot complete, one of them is the default posture of a brand-new bucket on a major cloud, and in the other not even the account root can delete the object.

There is a trap in how retention gets expressed, too. One popular table format counts its retention in commits rather than in days, which means the same unchanged configuration erases within hours on an hourly pipeline and takes ten days when someone moves that pipeline to nightly. No setting changed. No alert fired. The SLA moved by a factor of twenty-five. The general lesson outlives the specific product: any floor denominated in something other than time will drift against a deadline that is denominated in time, silently.

The money compounds quietly alongside it. Superseded object versions that no rule ever expires do not plateau, they accumulate, so the carry roughly triples between year one and year two on a petabyte-scale table. The fix is a few lines of JSON, and it is the cheapest line item in the whole talk.

None of this requires you to take our word for it, because the useful part is arithmetic rather than measurement. There is a two-term check you can run against your own tables during the session, from numbers you already know, that tells you whether your window is reachable at all. If it fails, no compaction policy, no table format and no delete mode will rescue you, and knowing that before you spend a quarter tuning a scheduler is worth the twenty minutes. If it passes, we will show you what coordinated scheduling buys on top: pulling every stage forward against the deadline instead of running each on its own cadence took our median from 34 days to 20, at a fraction of the rewrite budget the naive policy needed. We will also tell you what it costs, because it is not free.

We are explicit throughout about what was executed and what was not. The Iceberg chain was measured on real tables. The Delta and Hudi chains are read off their documentation and labelled as such. The cost figures are a model with its assumptions on the slide. Workloads are synthetic and time is virtual, so the day counts are lower bounds; at real scale the jobs take longer, not less.

This is for anyone who operates a lakehouse and has ever been asked to prove that a deletion actually happened: data platform and infrastructure engineers, the people who own compaction and vacuum schedules, and anyone whose name is on a data-retention commitment. You do not need to have read the papers. You do need to have shipped something to production and cared what happened next.

You will leave with a decomposition of where your own erasure latency actually goes, a way to measure residency on your buckets instead of inferring it, the feasibility check to run before touching a scheduler, and four things to look at on Monday morning. One of them takes about ten minutes and removes a liability that currently has no upper bound.

A delete is a promise about visibility. Erasure is a promise about bytes. Only one of them is scheduled by your table format.

Agent Traces are the new clickstream

Agent traces grow with the square of session length, because the prompt at step k is the whole conversation so far. On 400 public agent runs that is 36× the bytes of storing each message once. Stored three ways on Parquet and queried with DuckDB, typed columns beat raw JSON 91× on aggregates, while a message table keyed by hash replays 3.9× faster on half the disk and lets prompt text expire before the ids.

Who Needs a Warehouse When You've Got a Lakehouse?

The data scene has seen a big change in recent years, moving from old-fashioned data warehouses to more easy and strong lakehouse architectures. This switch means not just a technical advancement but a basic rethinking of how groups keep, handle, and get value from their data assets.
The traditional data warehouse worked well with structured data because there was a defined schema, consistency, and strong BI performance. The high costs and inflexibility of such warehouses meant that they were not suitable for handling unstructured data- much less a large volume of such unstructured data.
This is what gave birth to the data lake: an inexpensive place to store enormous volumes of highly variegated datasets-with schema-on-read flexibility. It then often turned into the "data swamp" due to quality issues, analytical performance shortcomings, and governance issues.
Then came the lakehouse architecture, a brilliant amalgamation that took unto it the strengths of both warehouse and lake.

Quicker Analytics : Self Serve Analytics to the rescue

As an Architect evaluating our analytics transformation roadmap, I've identified self-serve analytics as the critical accelerator for our enterprise data strategy. Our current centralized BI bottleneck creates an unsustainable multiple sprints average insight delivery timeline – completely incompatible with modern business velocity.
The proposed architecture implements a three-tier semantic modeling approach:

-- Core Data Layer: Leveraging our lakehouse medallion architecture with materialized views on Gold datasets, structured through domain-driven design principles
-- Semantic Modeling Tier: Implementing metric stores with SQL-based abstraction layers to decouple business logic from physical infrastructure
-- Visualization/Exploration Layer: Deploying governed tools supporting both SQL-fluent analysts and business users requiring GUI interfaces as playground for the data

Performance benchmarks from our POC demonstrate 95% reduction in time-to-insight, with 78% of previously centralized report requests now self-serviced using tools like Sigma, ThoughtSpot. Data mesh principles have been incorporated for domain-oriented ownership, while ensuring central governance through automated quality controls.

Data Observability and Reliability Engineering in a Real Time world

Data observability and reliability engineering are rapidly emerging as foundational pillars in modern data engineering and MLOps, and ensuring that data pipelines are robust, trustworthy, and capable of supporting critical business operations is imperative.
Data observability is the comprehensive ability to monitor, track, and analyze data as it moves through pipelines, providing real-time insights into data health, quality, and system performance
. It goes beyond traditional monitoring by offering a holistic, proactive approach to identifying and resolving issues before they impact downstream analytics or machine learning models
Data Reliability Engineering focuses on ensuring that data is consistently accurate, available, and dependable over time
. It leverages observability tools and practices to maintain high standards of data quality and system uptime, often borrowing principles from Site Reliability Engineering (SRE) such as Service Level Objectives (SLOs) and error budgets.

Walkthrough of Medallion : Why Organizations Need the Medallion Architecture

The medallion architecture is a data organization framework that has become crucial for organizations implementing data lakehouses. Here's why it's so valuable:
Structured Data Quality Management
The medallion approach (typically using Bronze, Silver, Gold layers) provides a systematic method to progressively improve data quality. Organizations can maintain raw data while ensuring downstream analytics use only validated, transformed data.
Clear Data Lineage
By organizing data through distinct processing stages, organizations gain transparent data lineage. This makes it significantly easier to trace how data flows through the system, troubleshoot issues, and satisfy regulatory compliance requirements.
Optimized Performance
The medallion architecture enables performance optimization at each layer. Organizations can structure their Gold layer for query performance, while maintaining Bronze layers for completeness and Silver for transformation logic.
Simplified Access Management
Different user groups require different data access levels. The medallion approach allows organizations to implement granular security policies—data scientists might access Silver data, while business analysts only work with curated Gold datasets.
Workload Isolation
Organizations can isolate intensive data processing jobs by layer, preventing resource contention. ETL processes on Bronze data won't impact analysts querying Gold datasets.
Accelerated Time-to-Insight
By providing pre-processed, validated data in the Gold layer, organizations dramatically reduce the time analysts spend preparing data, allowing them to focus on extracting insights instead.
Future-Proof Architecture
As data requirements evolve, organizations can adapt each layer independently without disrupting the entire pipeline, providing architectural flexibility for changing business needs.
The medallion architecture isn't just a technical implementation detail—it's a strategic approach that helps organizations balance data governance, performance, and accessibility in their lakehouse environments.

YouTube session and local Data Chapter

SQL Indexes - Boon or Bane?

SQL indexes are a powerful tool for optimizing database performance, but their effectiveness depends on platform-specific strengths and trade-offs. In Azure SQL, automated features like index tuning and columnstore indexes streamline analytical workloads, while managed maintenance reduces fragmentation risks. For PostgreSQL, flexibility shines with specialized indexes (e.g., GIN for JSONB, BRIN for time-series) and partial/expression-based indexing, enabling tailored optimizations. Both platforms enforce data integrity via unique indexes, and read-heavy systems benefit significantly. However, Azure’s automation can lead to unintended index drops, and columnstore indexes require partitioning discipline. PostgreSQL demands manual upkeep (e.g., VACUUM for bloat) and risks suboptimal plans without proper composite index design.

The downsides center on write overhead and cost. Azure SQL’s indexing increases DTU consumption and storage costs, especially in geo-replicated setups. PostgreSQL’s MVCC model causes index bloat, impacting distributed systems like Citus. Over-indexing in either system inflates storage: Azure’s tiered pricing penalizes excess, while PostgreSQL’s self-managed flexibility still demands cost-awareness. Ultimately, indexes are a boon when aligned with platform capabilities (e.g., Azure’s analytics focus, PostgreSQL’s data-type diversity) but a bane if applied generically without workload analysis and maintenance planning.

Intermediate (300) level : New Session