Session

AWS Modern Orchestration: Moving from Manual DAGs to Visual Workflows

In complex data analytics & automation projects, orchestrating dependencies across multiple accounts often leads to "DAG fatigue"—where engineering teams spend more time maintaining Python scripts in Apache Airflow than improving data quality. As data volumes grow, the overhead of managing infrastructure for these workflows becomes a bottleneck for agility.

This session provides an advanced look at the next generation of orchestration using Amazon SageMaker Unified Studio and Amazon MWAA Serverless. We will demonstrate how to replace manual orchestration code with a visual workflow experience that provides production-grade reliability without the operational burden.

In this session, learn how to:

- Automate Workflow Conversion: Technical details on how visual definitions are converted into Airflow-compatible definitions for MWAA Serverless.

- Implement Advanced Sensors: Using S3 Key Sensors and Glue Job Operators to create event-driven, "reactive" data pipelines.

- Manage Cross-Tool State: Handling state transitions between Visual ETL cleansing jobs and JupyterLab machine learning notebooks.

The Payoff: Attendees will leave with a practical framework for building collaborative, serverless orchestration environments that bridge the gap between data engineering and AI development.

Md. Mostafa Al Mahmud

AWS Community Builder, Senior software engineer, solutions architect, public speaker, and technical writer.

Bochum, Germany

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top