Azure Databricks has been around for a while now, and Apache Spark since 2012. You've watched a couple of demos, got a brief overview of what Databricks does and you've got a rough idea of where it fits in… but where do you go from there?

This session is that next stop. We'll start by taking a deeper look inside the spark engine, understanding what makes it tick and how it talks to data. We'll then break down some of the key features that come together to build the kind of data processing task that's changing how we think about ETL.

We'll be looking at:
• RDDs
• Schema Inference
• Metadata Management
• Parameterisation using Widgets
• Integration with ADF

If this is your first foray into Spark or Databricks, it'll be a bumpy ride!