The most uncomfortable meeting we ever sat in was with finance. They had pulled cloud spend data by team and wanted to understand why our data platform costs had tripled in eighteen months while the business metrics it supported had grown maybe thirty percent. We had good answers for some of it. For the rest, we had to admit we'd made engineering decisions that optimized for velocity and never thought about what they'd cost at the scale we eventually hit.
That meeting led to a cost audit that was, honestly, humbling. Full table scans on partitioned datasets running dozens of times a day. Glue jobs provisioned for peak load running constantly at ten percent utilization. Kinesis shards sitting idle because someone estimated high and nobody revisited it. S3 objects that should have moved to Glacier eighteen months ago still sitting in Standard storage because the lifecycle policy was on the roadmap and never made it to the sprint.
This talk isn't about being cheap. It's about being deliberate. At petabyte scale, the gap between an engineered cost posture and an accidental one is measured in hundreds of thousands of dollars annually. We'll share the cost attribution framework we built to actually understand where money was going—mapping specific pipeline components to specific spend across compute, storage, data transfer, and API calls, so engineers could see the financial impact of their architectural choices.
We'll cover the optimizations that moved the needle most: right-sizing EMR and Glue job configurations, S3 tiering and lifecycle automation, Kinesis shard management, and the Redshift Spectrum versus Athena decision tree that most teams get wrong because they're thinking about data volume instead of query frequency. We'll also talk about the behavioral patterns that quietly inflate costs and are almost never caught in code review.