We've been using Kolla-Ansible in the Workday Private Cloud for more than 5 years. We have been building and operating our clusters in a nearly air-gapped environment. Until recently, we have been using an internal automation tool to operate more than 200 OpenStack clusters.

This year, we started evaluating the changes required to switch to Ansible AWX to operate the clusters. This allowed us a rare moment to re-evaluate our orchestration architecture from first principles.

Whether you operate one cluster, or 200 clusters with Kolla-Ansible, this talk will be of interest to you. We will cover how we test our tools in CI, how our CD rollout works and how we handle all those pesky low chance failures that become important at scale.