Session

Feature Engineering at Scale: Apache Spark and Delta Lake on Azure Databricks

Feature engineering pipelines that work fine on a laptop-sized dataset often fall apart at production scale - data skew, slow joins, and reprocessing costs pile up fast. This talk walks through building scalable feature engineering pipelines using Apache Spark and Delta Lake on Azure Databricks, covering partitioning and join strategies that avoid common performance traps, using Delta Lake's versioning and time travel for reproducible feature sets, and structuring pipelines so MLOps and data science teams aren't fighting over the same tables. I'll share concrete lessons from building and debugging these pipelines in production, including where Spark's lazy evaluation quietly causes reprocessing you didn't intend.

Jubin Soni

Senior Software Engineer, Yahoo Inc

San Francisco, California, United States

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top