Outshift Platform team at Cisco supports many incubation projects that are at various stages of development life cycle. We implemented a multi-tenant Kubernetes environment with a standard namespace isolation for applications to run.

One of our core goals in this multi-tenant environment is to dynamically inject application secrets from an external secrets management system like HashiCorp Vault into Kubernetes secrets, while preserving self-service capabilities of the developers to manage their projects independently. An individual team application should only be allowed access to secrets that belong to their team.

Forementioned requirements posed a unique challenge in securely injecting secrets into various namespaces without compromising the least privilege requirement across team boundaries. Also, such a system must be consumed by developers in an understandable and automated way.

This talk will demonstrate how the platform team overcame this challenge by leveraging a combination of the External Secrets Operator, HashiCorp Vault Kubernetes authentication and Vault policies. It will also demonstrate how the developers are able to easily specify custom resources with secrets paths scoped to their team that will be injected into target Kubernetes secrets. These custom resources are bundled along with application helm charts and deployed via standard gitops methods.

Secrets injection from external secret managers in multi-tenant Kubernetes has become a common practice in many organizations. The mechanics of how this is achieved across organizations is not standardized. On one extreme, the developers manually deploy application secrets and avoid setting up proper automation. Some implementations often allow application teams wide access to secrets than they are supposed to have.

Tools used for secrets injects are opaque to developers which leads to more confusion and frustration. In summary, self-service and automation of secrets injection is lacking in many organizations.

This talk will a methodology that has achieved clarity in automation, good developer success and is currently in practice in a large organization serving various independent projects. This is methodology is offered a as a model for implementing secrets injection to other organizations with similar needs.