A key requirement for data pipelines is that they produce high quality data.

At Next Insurance we use quality gate tasks as circuit breakers within our workflows, but we were struggling to measure their effectiveness. There's no "effectiveness" indication in Airflow, so in order to evaluate our workflow's quality, we built a graph analysis tool to analyze its quality gates.

In this talk, we will show how our tool drives quality by calculating a score for each workflow, giving us a bottom line metric for tracking improvements, analyzing the flow, highlighting gaps in coverage and providing statistics on gate effectiveness.

I'll cover the methodology behind the tool as well as the pitfalls and edge cases, and show how we integrated the tool into our CI/CD systems.

By applying these concepts to your own processes, you too will be able to improve visibility into your workflow's quality and set KPIs for future improvement.