Argo Rollouts makes Progressive Delivery easy to adopt, but some times things do not work as expected. Are the steps correctly set? are the analysis metrics right for the workloads?

At Adobe Experience Manager we deploy over 10k customer services to Kubernetes. Changes can occur multiple times per day both internal and from code. A new feature can work fine for 99% of customers but still affect the other 1%, and detecting this just from tests is costly. Enter Argo Rollouts, which allows deploying new versions to a subset of users before rolling them to the totality of the users, and rolling them back if not matching some key metrics, using techniques like canary deployments.

We will show our learnings deploying Argo Rollouts to manage over 10k workloads using canaries, how do we balance speed and safety for our customers, and some of the issues that we have faced when adopting it.