Shubham Raj
Senior Engineer, Qualcomm | Apache Airflow Committer
Bengaluru, India
Actions
Shubham Raj, a Senior Engineer at Qualcomm and an Apache Airflow committer, is a full-stack engineer who builds semiconductor workflow orchestration infrastructure for the Snapdragon CPU team and has experience building data platforms on Kubernetes in private cloud environments.
He enjoys sharing knowledge and actively learning from the open-source community.
Links
Area of Expertise
Topics
DAGs Move Robots: Closed‑Loop Orchestration for Silicon Validation Labs with Airflow
What if your Airflow DAG could orchestrate robots, thermal chambers, and silicon tests, not just code?
Silicon validation labs rely on scarce, stateful physical resources: robotic handlers, DUT boards, thermal/power
systems, instruments, and shared hardware queues. Teams often coordinate these via spreadsheets and ad hoc
reservations, causing contention, idle gaps, conflicts, poor observability, and slow triage.
This talk presents a closed-loop orchestration model where Apache Airflow is the control plane for a software-defined
validation lab. A central DAG coordinates robotic handling, thermal/power setup, stress and performance runs, and
parametric characterization on hosts connected to silicon. It continuously ingests hardware health, measurements, and
test outcomes, then feeds results into AI-assisted analysis to choose the next physical action: refine parameters,
schedule follow-up experiments, or trigger mitigation.
Using Edge workers on dedicated lab machines, we replace manual coordination with reliable, auditable orchestration.
The same pattern extends beyond silicon to robotics labs, device farms, and other cyber-physical environments.
Airflow 3’s Trigger UI: Evolution of Params
Are you looking to build slick, dynamic trigger forms for your DAGs? It all starts with mastering params.
Params are the gold standard for adding execution options to your DAGs, allowing you to create dynamic, user-friendly trigger forms with descriptions, validation, and now, with Airflow 3, bidirectional support for conf data!
In this talk, we'll break down how to use params effectively, share best practices, and explore what's new since the 2023 Airflow Summit talk (https://airflowsummit.org/sessions/2023/flexible-dag-trigger-forms-aip-50/). If you want to make DAG execution more flexible, intuitive, and powerful, this session is a must-attend!
Overcoming Custom Python Package Hurdles in Airflow
DAG Authors, while constructing DAGs, generally use native libraries provided by Airflow in conjunction with python libraries available over public PyPI repositories.
But sometimes, DAG authors need to construct DAG using libraries that are either in-house or not available over public PyPI repositories. This poses a serious challenge for users who want to run their custom code with Airflow DAGs, particularly when Airflow is deployed in a cloud-native fashion.
Traditionally, these packages are baked in Airflow Docker images. This won’t work post deployment and is super impractical if your library is under development.
We propose a solution that creates a dedicated Airflow global python environment that dynamically generates the requirements, establishes a version-compatible pyenv adhering to Airflow’s policies, and manages custom pip repository authentication seamlessly. Importantly, the service executes these steps in a fail-safe manner, not compromising core components.
Join us as we discuss the solution to this common problem, touching upon the design, and seeing the solution in action. We also candidly discuss some challenges, and the shortcomings of the proposed solution.
Sketching Pipelines using DAG Authoring UI
Cloudera Data Engineering (CDE) is a serverless service for Cloudera Data Platform that allows you to submit various Spark jobs and Airflow DAGs to an auto-scaling cluster.
Running your workloads as Python DAG files may be the usual, but not the most convenient way for some users as it involves a lot of background around syntaxes, the programming language, aesthetics of Airflow, etc.
The DAG Authoring UI is a tool built on top of Airflow APIs to allow one to use a graphical user interface to create, manage, and destroy complex DAGs. The DAG authoring UI will give one the ability to perform tasks on Airflow without really having to know DAG structure, Python programming language, and the internals of Airflow.
CDE has identified multiple operators to perform various tasks on Airflow by carefully categorising the use cases. The operators range from BashOperator, PythonOperator, CDEJobRunOperator, CDWJobRunOperator
Most use cases can be run as combinations of the operators provided.
Airflow Summit 2025 Sessionize Event
Airflow Summit 2024 Sessionize Event
Airflow Summit 2023 Sessionize Event
Shubham Raj
Senior Engineer, Qualcomm | Apache Airflow Committer
Bengaluru, India
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top