From Model to Production: Efficient LLM Deployment with Kubernetes

Unlike deploying a simple web application on Kubernetes, serving AI/ML workloads involves a series of complex decisions to ensure they run efficiently and reliably in production. These decisions include things like understanding the model architecture, selecting the right hardware, choosing the appropriate inference backend or server, implementing effective auto-scaling, benchmarking the models’s performance, and optimizing for scaleable and reliable deployments.

In this session, we’ll walk you through the key steps—from having a trained/fine-tuend model to running it on Kubernetes in a scalable, cost-efficient, and high-performance manner. You’ll leave with a clear understanding of the practical architectural choices required to build and operate production-grade inference workloads on Kubernetes.

Audience Takeaways:

1. Understand the core architectural decisions that impact how AI/ML workloads run on Kubernetes in production.
2. Gain practical guidance for deploying AI/ML workloads in a scalable, efficient, and optimized way.
3. Learn how the Kubernetes ecosystem and its specific features simplify running production-grade inference at scale.

Navigating the Observability Landscape: Choosing the Right Pattern for Your Kubernetes Stack

As Kubernetes adoption grows, organizations face the challenge of selecting the best observability stack to monitor and manage their complex cloud native environments. We'll explores the architectural patterns and considerations for choosing an observability stack that aligns with your organization's needs, skills, and budget.
We'll discuss the spectrum of observability solutions, ranging from fully managed solutions that eliminate the burden of scaling or handling complex maintenance requirements , all the way to self-hosted open source options that offer flexibility but require a wider skill set.
We'll discuss how Kubernetes might also facilitate the integration of third-party services directly into the user's stack, effectively blurring the lines between managed and open-source solutions.
Attendees will gain insights into the pros and cons of various observability architectures, covering management aspects, data residency and privacy, trade-offs around cost implications.

Deploying Apps Like Pros: Team-Driven Multi-Cluster Strategies with Crossplane, ArgoCD, and EKS

With the open flexibility of Kubernetes and its ecosystem, there are many ways to deploy applications to Kubernetes. We need to design our deployment process to support both platform and application teams. In this tutorial, we will cover deployment practices we’ve seen implemented by end-users, and will stand on the tradeoffs of using one deployment method or the other. You will have a hands-on experience for deploying applications into Kubernetes, leveraging some of the common practices such as IaC, GitOps, and the bridge between them. You will also experiment with practices for provisioning backing resources using Crossplane and ArgoCD. Let's make deploying applications a breeze!

I can get some SaaS-isfaction - Crafting a SaaS Multi-Tenant Architecture on Kubernetes

Building SaaS product require you to design for tenant isolation, routing, deployment strategies, and other strategies for tenant management. Tenant provisioning should support different tenant configurations with the different strategies mentioned above, as well as supporting different deployment types (all-at-one, rolling-upgrade, staggered) for the provisioned tenants. It should also accommodate multiple teams working on different apps, all of them with different design and deployment method for different types of tenants.

In this session we will share our experience in designing and building multi-tenant solution based on Kubernetes and its ecosystem. We will dive into the tenant configuration process, and to application specific deployment for all the tenants in the system. We will demonstrate our implementation of using Kubernetes alongside with GitOps using Argo CD for tenant provisioning, and Helm for templating app configuration for different tenant tiers

Multi-Cluster Management Made Easy: Leveraging IaC and GitOps for Accelerated Application Deployment

Platform teams focus on enabling other teams in the organization. They are responsible for providing the right tooling to Application teams, allowing them to focus on building new functionality. Using Infrastructure as Code (IaC) tooling and a GitOps-based delivery model for Kubernetes cluster provisioning and configuration greatly accelerates application deployment by empowering application teams to independently manage their deployment process. It also enables managing multiple clusters consistently and efficiently across multiple environments.

In this session learn how to create an automated IaC blueprint configuration to build your Kubernetes clusters across multiple environments. You will also learn how the combination of IaC and GitOps enables cluster multi-tenancy, security controls, and increases deployment velocity for application teams.

Scaling for Success: Building modern Architectures and Developer Acceleration

Decisions made at the beginning of building a technical product have an impact on its future scale, delivery velocity, and the way to build new services. This session explores modern architectures, while focusing on containers technology, to guide teams in making the right choices. It also examines how you can design your organization and your technology for constant change. We’ll discuss which technology choices are crucial to getting started and, more importantly, which choices can be deferred to a later stage as the project scales and evolves. Combined with DevOps and Platform engineering practices, we will discuss what needs to be done to enhance developer productivity. Drawing on insights from our past experience working with a wide range of customers, we will explore the journey of projects, from initial technology decisions to scaling challenges, and share key lessons with examples.

Kubernetes Deployment and Management: From Platform to API

When organizations adopt Kubernetes, they want to improve productivity, developer experience, and consistency.Kubernetes is used by some customers as a platform, and IaC tooling is used to provision backing services. Others are trying to standardize Kubernetes as a centralized API to manage both application components: runtime and infrastructure.Platform teams focus on building deployment abstractions to reduce cognitive load on developers and facilitate deployment of fully configured applications.
The session will explore different options for deploying infrastructure and runtime components of the application and showcase open source tools such as Kubevela and Crossplane that enable those deployments. We will conclude with a demonstration to deploy the application components (runtime and infrastructure services) leveraging the Kubernets API as an abstraction layer.

Kubernetes is a the commonly used platform for deploying containerized applications. It’s flexible and customizable with many configuration options. However, with such large ecosystem, developers wants to focus on what’s really matter for them, which is delivering business value through deploying applications. To address that need platform teams focus on building abstractions , to reduce the cognitive load from the developers, and providing mechanism to easily deploy fully configured application.
In this session we will cover the different options of achieving such abstractions , from self-developed systems, to open-source projects such as ACK, Helm, or Crossplane. We will discuss the design consideration when implementing such abstractions , and share some pitfalls around creating leaky abstractions.

Any Accelerator, Any Time: Mastering Cost-Efficient GenAI Inference at Scale

As generative AI adoption accelerates, organizations face mounting challenges in delivering cost-effective inference at scale. This session demonstrates how to build an intelligent, multi-accelerator inference platform that automatically optimizes for cost and performance. We'll show how to orchestrate popular open-source models like Llama, Mistral, and Stable Diffusion across heterogeneous compute resources - from NVIDIA GPUs to AWS Inferentia - using Kubernetes as our foundation. Through practical examples, attendees will learn how to implement dynamic workload scheduling that responds to real-time demand using KEDA and Prometheus metrics. We'll explore how Karpenter enables automated provisioning of the right compute resources at the right time, and how frameworks like Neuron and Triton can be leveraged for optimal performance across different accelerators. You'll leave with actionable insights on building scalable, cost-efficient GenAI inference pipelines that can intelligently adapt to varying workload demands while maintaining performance SLAs.

Tsahi Duek public speaking playlist
https://youtube.com/playlist?list=PLxuZS4zp_O2QkooK9cBkgYWr-5R3Y4mWC&si=_LoUPzA5ZrwUpVcd

Decisions Fatigue: A Practical Guide to Application Ready Kubernetes Cluster

The Kubernetes ecosystem offers an unprecedented array of tools, platforms, and solutions for container orchestration and cloud-native development. While this breadth of choice allows end-users with flexibility and customization options, it also presents a significant challenge causes decision fatigue.

This talk covers the key decision points in building application-ready Kubernetes, from cluster deployment tools to autoscaling configurations. Based on real-world examples, attendees will learn:

* Recommended project structure for cluster management
* Deployment tooling selection
* Autoscaling optimization
* Application configuration standardization
* Observability stack configuration

Each decision covered includes trade-off analysis, and implementation guidelines derived from production experiences, enabling teams to accelerate their path to a production-ready Kubernetes platform.

At the end of this talk you would be able to effectively deploy reproducible, well-configured, and application-ready Kubernetes clusters.

Benefits to the ecosystem:

Provisioning a cluster is not just its control-plane and nodes. To make a cluster ready-to-use for actual application deployment, users have to decide which tool to use to enable specific capabilities, such as: application networking, autoscaling, security controls, storage integration, and many more. Those tooling can be anything from CNCF project such as: KEDA for autoscaling, Cilium for networking, ArgoCD for GitOps delivery, or even vendor provided addons.

This talk accelerates Kubernetes adoption and promotes the use of complementary CNCF projects by providing clear, practical guidance for deploying cluster capabilities, helping end-users minimize decision-making while leveraging the broader cloud-native ecosystem.

At the end, attendees will leave with a good understanding of the foundational tooling they should have, and when they should explore other implementations