Session
Cost-Aware Data Engineering: Lessons from Building at Scale on AWS
Our data platform costs tripled in eighteen months while business value grew thirty percent. Finance noticed. The audit that followed was humbling: full table scans on partitioned datasets running dozens of times daily, Glue jobs provisioned for peak load running at ten percent utilization, Kinesis shards sitting idle since the original estimate, and S3 objects in Standard storage because lifecycle policies never made it to the sprint. This session is a practical guide to cost-aware AWS data engineering drawn from that audit. We start with a cost attribution framework that maps pipeline components to specific spend categories — compute, storage, data transfer, API calls — so engineers can see the financial impact of their architectural choices before they compound. The bulk of the session covers the optimizations that moved the needle most: right-sizing EMR and Glue job configurations, S3 lifecycle tiering automation, Kinesis shard management, and the Redshift vs. Athena decision tree most teams get wrong because they evaluate data volume instead of query frequency. The 60-minute format gives us time to go deep on the Spark join and partitioning anti-patterns that quietly inflate costs in ways code review never catches, and to close with a governance model — cost tagging, budget alerting, ownership assignment — that keeps spend visible and accountable before it becomes a difficult conversation with leadership.
Anusha Kovi
Business Intelligence Engineer, Amazon
Seattle, Washington, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top