
Pankaj Koti
Senior Software Engineer @ Astronomer.io
Pune, India
Actions
I am a software engineer with almost 9+ years of Software Industry experience and currently working with Astronomer.io Previously, I have worked with various small startups as well as established organisations in positions such as Backend, FullStack and Data Engineer in India and Berlin, Germany. I have been using Airflow since 2018 and love all of it, especially its strong community.
Links
Area of Expertise
How to contribute deferrable operators to Airflow OSS. Why use them?
Much work has already been done and invested in Airflow to support deferrable operators. The talk aims to give a brief introduction to how to contribute deferrable operators to Airflow; aiming to increase the adoption & usage of deferrable operators and thus, show how Airflow is cost-effective when it comes to using operators. Also, it aims at inviting users to contribute deferrable mode support to existing providers in Airflow by walking them through an Airflow PR showing such an implementation.
Many users are still seen reporting over various platforms like the Airflow Slack channel, StackOverflow, etc. that their limited worker slots often remain occupied while the tasks are waiting on something & thus living limited resources for executing other due tasks that could execute in parallel while the currently running tasks are waiting for some event/action. The talk would help solve this problem and also show an experimental study showing the cost savings by using deferrable operators.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top