Maintenance 101: How to randomly delete 230 database objects in production (Case study)

This talk will tell the story of how a scheduled production update for a customer with no expected downtime turned into a 16-person OnCall session at 3 AM and a 30-person session at 10 AM. More importantly, it will outline how to move forward without losing the customer.

As much as one would like to think that the latest patch has been thoroughly tested and all risks have been assessed, deployment to production is almost like starting it all over again. Maybe your test environment is 1:1 with production, but ours wasn’t. 12 clusters and 600 MariaDB and 700 Redis databases with a Crossplane control plane are effortful enough as a productive system. Therefore, our test environment was just a fraction of it with 1 cluster and 25 databases.
This partly tech and partly business talk will cover the importance of proactive behavior, measuring what you cannot see, maintaining the trust of the customer, and obviously meaning different things to different people.

30 min case study

How we used Crossplane for the things we should not have

Frameworks are meant to be tweaked, that is just a part of being an engineer. Even if a new cloud native, open source framework is exactly what you are attempting to build.
This talk tells the story of what Crossplane is and what we wanted it to be. It highlights the need for the emerging Crossplane V2, and why they are different. It also illustrates a real life story when bending leads to breaking, with losing 230 production databases for a customer during maintenance. The silver lining - we kept the customer, but changed our approach.