Multi-tenant Data Pipeline Orchestration

Designing data pipelines with a single codebase that is executed on multiple tenants, data sources, and runtime parameters, introduces unique challenges and opportunities.

This session will review industry-leading frameworks such as Airflow, Beam, and Dagster, discussing their strengths and trade-offs in handling multi-tenant orchestration.

We will cover methods for identifying small but impactful regressions due to code updates or external data variations, as well as practical insights into choosing the right framework based on scalability, monitoring capabilities, and ease of integration. Attendees will gain insights into the often-overlooked challenges of multi-tenant orchestration and practical strategies for addressing them.