Databricks is not just notebooks and Spark SQL - it’s also a powerful platform for automating and orchestrating data processing workflows. During this session, we’ll show how to tame jobs, tasks, and triggers so they run when they should - and do what they’re supposed to – without anyone having to wake up at 3:00 a.m.

Using real examples, we’ll walk through how to design workflows that not only work but are also maintainable and scalable - from simple DAGs to more complex task dependencies. We’ll also compare several job triggering methods: manual, scheduled, event-based, and webhook-based. Expect some coverage of retry logic, inter-job dependencies (not just within a single workflow), notifications, and common pitfalls (“why didn’t this run?!”).

And to top it off - best practices, lessons learned from production deployments, and a bit of dark humor from the life of a data engineer.

After this session, you’ll:
- know which orchestration mechanisms in Databricks actually work in the real world (and which only look good on slides),
- understand how to build automation without fearing late-night pings from your monitor,
- and - perhaps most importantly - regain control over whatever is quietly running “somewhere in the background.”