Poonam Kucheriya
Senior Cloud Engineer at AWS
Actions
Poonam has over 14 years of software experience as a developer, Data Specialist, and Cloud Engineering in both medium and large corporations and is the AWS Support Experience Owner responsible for Amazon Managed Workflows for Apache Airflow (MWAA).
Links
Let the air flow : Capacity Planning in Apache Airflow
When setting up an Airflow cluster, it is possible to face issues with resource overloading or over-provisioning. While Airflow offers numerous configuration options to flow-control workload and prevent peaks, identifying the most effective option during congestion can be challenging.
To address this challenge, a capacity model is introduced to assist Airflow users in efficiently planning and managing their clusters. This model comprises three components: Workload, Flow-control, and Resource. This session aims to address various questions related to this model, such as
- what kind of workload runs on schedulers and workers,
- how scheduler workload is shared in a high-availability model,
- and how to configure Airflow to prevent scheduler overloading.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top