Martijn Beenker
Senior Data Engineer at Team Rockstars Data
Amsterdam, The Netherlands
Actions
Senior data engineer and data platform engineer @ Team Rockstars Data. Certified Azure, Python and Kubernetes developer who has worked in the financial and telecom industry. During his day-to-day work Martijn places strong emphasis on developer happiness, convinced that happy devs make better software and systems.
Area of Expertise
Topics
Mistakes we made while scaling Azure Kubernetes Service to enterprise level
Rabobank the Netherlands is building an enterprise grade data mesh. And within that mesh, our team was tasked with implementing Airflow as the defacto orchestration tool. For hundreds of teams. Running on Azure Kubernetes Services.
Looking back over the last two years, we can finally say we did it. The application is now hosting over 50 teams, running smoothly, linking data delivery streams with data consumption streams. And our team has become a sort of Airflow Center of Excellence. However, we did not get to this point without making our fair share of mistakes.
This session will cover every step of our journey, including: struggling with PostgreSQL db backend (single server), Azure Virtual Network design, using Kubernetes autoscaler and handling default application settings.
Halting Bad Data in Its Tracks: Real-time Data Quality Gates in Microsoft Fabric
Halting Bad Data in Its Tracks: Real-time Data Quality Gates in Microsoft Fabric
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top