Accidentally building a golden path for 100+ devs at a large bank

Automating a terrible process doesn't make it any less terrible. As DevOps and Platform engineers, we see this often. I worked on an app where deploying to production had a 60% chance of needing an emergency fix. This couldn’t go on. I automated the manual steps, but it didn’t solve the core issues.

The old process was scrapped, and a new one was built with input from many roles. Today, we’d call this the golden path. Building it was hard; getting teams to use it was harder.

This session shares lessons from building and adopting the golden path:
- Knowing when to improve or replace a process
- Highlighting the new process’s benefits
- Training and onboarding teams
- Managing edge cases
- Encouraging contributions while considering all needs
- Setting up guardrails to guide developers

Join to learn how to create and implement a sustainable process.

The target audience for this session is any platform or DevOps engineer responsible for building processes for their organization. The database deployments are a framing device. There is no specific technology requirements, and anyone with 1+ years of experience should benefit.

What is this DevEx and Platform Engineering I keep hearing about?

Over the past few years, a seismic shift has occurred in the technology landscape.
It seems like everyone is moving traditional Windows and Linux application hosts to modern solutions such as Containerization, Kubernetes, PaaS, and Hybrid Cloud models. That alone is complex enough, but the success of DevOps and the "you build it, you run it" mindset has added a new layer of complexity. Each self-managing feature team is forced to make numerous micro-decisions in transforming their applications.

When updating a few applications, that is not a big deal. However when replicated across multiple teams within the same organization, it becomes complex quickly. This complexity, often underestimated, has paved the way for the emergence of Platform Engineering—a discipline that does not seek to supplant DevOps but to augment and streamline its application across a growing enterprise. At its core, Platform Engineering concentrates on enhancing the Developer Experience (DevEx), which is pivotal for leveraging the full potential of DevOps at an organizational scale.

- Understanding the key attributes that make a platform engineering team successful in navigating the challenges of digital transformation and DevOps integration.
- Emphasizing the significance of prioritizing developer experience and treating processes as products in their own right. This approach ensures a seamless and efficient workflow for feature teams.
- Examining successful patterns as well as common pitfalls in platform engineering and developer experience, providing valuable insights to help organizations make informed decisions.
- Delving into the strategies for constructing golden paths that streamline development processes, making it challenging for developers to resist or deviate. This includes practical approaches to aligning teams with organizational goals.

This is an introductory talk for platform engineering. The target audience of this talk is any developer or DevOps engineer who is undergoing a digital transformation and needs to quickly "level-up" a lot of developers.

From EC2 to Kubernetes: Takeaways from a brownfield migration

From 2018 to 2019, we migrated our SaaS product from a conventional setup of 1800 EC2 instances to a dynamic and scalable architecture managed within Kubernetes Clusters.

This session will discuss the entire journey of our migration endeavor, from the nine-month conversion phase to the ongoing challenges encountered in the post-migration landscape. Topics discussed will be migrating existing customers, mitigating the impacts of noisy neighbors, devising efficient deployment strategies, and fine-tuning resource allocation to align with individual customer usage patterns.

Key Takeaways for Attendees:
- Insights into the rationale behind transitioning from EC2 instances to Kubernetes.
- Strategies for provisioning dedicated container instances to each customer while preventing interference from noisy neighbors.
- The rationale behind opting for managed services for our database and file storage needs.
- Challenges encountered during the transition from hosting on Windows to a Linux-based container environment.
- Best practices for routine maintenance tasks while respecting customers' maintenance windows.
- Navigating deployment challenges, leveraging "dogfooding" strategies, and implementing release rings.
- Techniques for dynamically adjusting CPU and RAM resources based on customer usage metrics.

The target audience for this talk is someone aware of kubernetes basics, such as nodes, containers, and namespaces, but not an expert. It is designed for anyone who has been a developer, operations, or platform engineer for with 2+ years of industry experience.

Rollback Strategies with Kubernetes

Kubernetes offers a powerful rollback feature for container versions, facilitating easy reversion by modifying the manifest file and applying changes. However, the automatic execution of rollbacks raises questions about feasibility and advisability.

Many containerized applications leverage relational databases for data storage, as transitioning from relational to NoSQL databases entails significant costs and time investments. Rollback attempts with relational databases pose inherent risks of data loss or corruption.

This session explores effective strategies for mitigating rollback risks in applications reliant on relational databases. We will explore a comprehensive database change strategy supporting multiple application versions. Furthermore, we will examine how Kubernetes empowers the adoption of advanced deployment patterns, enabling thorough testing of changes in Production environments before user accessibility.

Key Learning Points for Attendees:

- Streamline rollback processes by omitting database-specific steps in deployment procedures.
- Implement the expand/contract pattern to decouple database alterations from application updates.
- Embrace Blue/Green or "Staging" deployment methodologies to test changes in Production before end-user interaction rigorously.

By the end of this talk, attendees will gain valuable insights into fortifying their deployment pipelines and enhancing the resilience of relational database rollbacks within Kubernetes environments.

This session is designed for developers, operations, or DBAs who have at least 1 to 3 years of experience and are looking to adopt a rollback strategy.

Multi-Tenancy Architecture Lessons Learned

Architecting a multi-tenanted application is challenging. Decisions can impact developers and operations for the life of the application, which is measured in years, if not for over a decade.

Often, the hardest decision is which of the three common multi-tenancy architecture approaches is appropriate. A unique application and database per tenant, shared application and isolated database per tenant, or shared application and database. This decision is so challenging because there is no "one true approach." Each application's scenario is different.

I've witnessed the impact of those decisions firsthand. I've been exposed to each approach during my career as a software developer and architect. I spent over 20 years supporting six different multi-tenanted applications for companies working in advertising, telemarketing, oil, finance, government, and software.

In this session, we will walk through the lessons learned from working on those applications.

In this talk, attendees will learn:
- The pros and cons of each approach; unique application and database per tenant, shared application and isolated database per tenant, and shared application and database.
- When one approach is preferred over the other.
- How to handle unique configurations and requirements per tenant.
- The hidden costs and the long-term impacts.
- How advancements in tooling and platforms have made the unique application and database per tenant easier to support.

The target audience for this talk is architects, developers, and people in operations. The session is best for someone who has been in the IT industry for 1-2 years.

Kubernetes for the rest of us

Kubernetes, often perceived as a complex "black box," is a robust platform for container orchestration. This conference session is tailored for those who may not directly manage Kubernetes clusters but seek to understand its fundamental workings. This session aims teach attendees the basics on running a container in a Kubernetes cluster, particularly with an external database backend.

- Core Kubernetes Concepts: Explore essential concepts such as services, pods, nodes, clusterIPs, ingress controllers, secrets, deployments, and more. Gain a foundational understanding of how these components interact within the Kubernetes ecosystem.
- Deployment Strategies and CI/CD Pipelines: Unpack deployment strategies and dive into (CI/CD) pipelines. Understand the role of automation in managing containerized applications and discover where the emerging GitOps methodology fits into the Kubernetes landscape.
- Hands-On Learning Strategies: Equip yourself with practical knowledge by exploring ways to advance your understanding. Learn how to install and manage a Kubernetes cluster locally, providing a hands-on experience to solidify your comprehension.

By the session's end, participants will have a foundational grasp of Kubernetes concepts, insights into deployment strategies and CI/CD pipelines. Whether you're a developer, part of an operations team, or simply curious about container orchestration, this session simplifies Kubernetes, making its powerful capabilities accessible to all.

This session is designed for anyone who is aware of kubernetes and containers but isn't directly responsible for hosting or managing the cluster.

Don't ignore your users

Many DevOps, CI/CD, and Platform Engineering initiatives dream of deploying to production in the middle of the day. But is it practical for your app? Not every application is alike, nor is every user. I learned this the hard way, treating an internal app like it was Facebook. I pushed for faster production, blind to the impact on users.

- What happens when you change an app midday?
- When is the best time to deploy without disrupting systems?
- Training demands time and resources.
- Constantly explaining new features overwhelms users.

Join this session as we set realistic goals and pave smoother paths. We’ll focus on faster deployments without losing sight of user needs. Topics include:

- Setting priorities based on your users and app.
- Navigating company policies, regulations, and contracts.
- Knowing when "good enough" is better than perfect.

Let’s find the balance between speed and user satisfaction.

This session is designed for any DevOps or Platform Engineer focused on building processes and paved paths for developers. Minimal experience is required, as it will be discussing lessons learned.

How we progressively deliver changes to Kubernetes using Canary Deployments and Feature Flags

This is the case study of how we changed how we ship software.

With thousands of customers, each in their own Kubernetes container, deploying updates was tough. Off-hours schedules meant it took over 24 hours to push a new version. If something broke, we had to scramble. Canary deployments let us update small groups of customers at a time. We built a tool to stop rollouts fast when issues appeared, limiting the damage.

In the past, new features went to everyone at once. Rolling back wasn't an option. If something failed it'd leave customers stuck in the mess. Now, using OpenFeature, we hide new functionality behind feature flags. We release features to small groups, gather feedback, and test internally for weeks. If things go wrong, we flip the flag off and move on.

This two-pronged approach lets us avoid risky big-bang releases. We went from deploying every 10 days to every 4, with fewer than 1% high-severity defects. Most of these are resolved before customers notice them.

Stop Half-Deploying: Unlock Progressive Delivery and Rollbacks with Argo Rollouts and OpenFeature

This session dives into building a smarter progressive delivery pipeline for your applications. Argo Rollouts provides Blue/Green and Canary Deployments, which split traffic between versions and perform basic checks. But too often, this is where progressive delivery stops. Showstopping bugs might be found hours or days after deployments.

True innovation happens when you pair Argo Rollouts with feature flags like those from OpenFeature. OpenFeature extends feature flags beyond simple on/off toggles. OpenFeature enables targeting specific user segments, allowing controlled testing of new functionality over hours, days, or weeks. This approach lets you gather feedback from real users before a full release, reducing risks and improving outcomes.

When combined, Argo Rollouts and OpenFeature make rollbacks seamless. Argo halts traffic to unstable deployments, while OpenFeature disables faulty features for select users or everyone. Together, these tools ensure safe, efficient, and iterative innovation.

Focus on users, not traffic for canary based deployments

In today's tech landscape, canary deployments are more accessible than ever before. With modern tooling, it's simple to direct a portion of traffic to the latest code iteration. However, the conventional approach of traffic percentage allocation may overlook crucial considerations regarding user experience.

Canary deployments offer a robust strategy for minimizing deployment risks, but engineering teams encounter various challenges in their implementation:

- Ensuring backward compatibility in code and database modifications.
- Determining optimal timing for scaling up traffic exposure.
- Understanding the broader user impact when only a subset receives new features.
- Addressing post-deployment issues effectively.

This session delves into a user-centric perspective on canary deployments, aiming to achieve deployment objectives while mitigating user confusion. Attendees will gain insights into adaptable strategies applicable to diverse applications. Key takeaways include:

- Recognizing the limitations of traffic percentage-based strategies.
- Leveraging user segmentation by factors such as location, team affiliation, or job role.
- Implementing release rings tailored for Software as a Service (SaaS) providers.
- Offering opt-in mechanisms for users to trial new features.
- Embracing the expand and contract pattern for seamless database alterations.