Xiaodong Deng
Apache Airflow PMC member & Committer; Software Engineer at Apple
Actions
Xiaodong is currently working as software engineer in Apple ASE Data Pipelines team. He is also one of the committers (since March 2019) and PMC members (since December 2020) of Apache Airflow.
Other than coding & open source, Xiaodong also loves outdoor sports. He used to be an active trail runner, and finished races like Ultra-Trail Mt. Fuji (STY), TransLantau 50, Dalian 100, Canyons Endurance 50k, and more. He is also a P3 paragliding pilot.
Links
Create Your Custom Secrets Backend for Apache Airflow - A guided tour into Airflow codebase
This talk aims to share how Airflow's secrets backend works, and how users can create their custom secret backends for their specific use cases & technology stack.
Lessons Learned While Using Airflow as Open-Source Software
Apache Airflow is an essential piece of the data infrastructure for many organizations and has been largely adopted by data engineers across domains for orchestration. Due to its open-source nature, there are varied strategies to operate Airflow, resulting in different challenges. In this talk, we will explore common challenges when Airflow users operate it as an open source software, and the lessons learned. Such lessons should be applicable for operating other open source softwares as well.
Building in resource awareness and event dependency into Airflow
In this talk, we will explore how adding custom dependency checks into Airflow’s scheduling system can elevate Airflow’s performance.
We will specifically discuss how we added general upstream events dependency checking as well as how to make Airflow aware of used/available compute resources so that the system can better decide when and where to run a given task on Kubernetes infrastructure.
We’ll cover why the existing dependency checking in Airflow is not sufficient in our use case, and why adding custom code to Airflow is needed. We’ll cover the pros and cons with this approach.
Better Support for Using Multiple Namespaces with KubernetesExecutor
Airflow’s KubernetesExecutor has supported multi_namespace_mode for long time. This feature is great at allowing Airflow jobs to run in different namespaces on the same Kubernetes clusters for better isolation and easier management.
However, this feature requires cluster-role for the Airflow scheduler, which can create security problems or be a blocker for some users.
PR https://github.com/apache/airflow/pull/28047 , which will become available in Airflow 2.6.0, resolves this issue by allowing Airflow users to specify multi_namespace_mode_namespace_list when using multi_namespace_mode, so that no cluster-role is needed and user only needs to ensure the Scheduler has permissions to certain namespaces rather than all namespaces on the Kubernetes cluster.
This talk aims to help you better understand KubernetesExecutor and how to set it up in a more secure manner.
Xiaodong Deng
Apache Airflow PMC member & Committer; Software Engineer at Apple
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top