Suchita Venugopal
Senior Machine Learning Engineer, Pagerduty
Vancouver, Canada
Actions
Suchita Venugopal is a Senior Machine Learning Engineer at PagerDuty, working at the intersection of data engineering, LLMs, and agentic systems.
Her current focus spans two areas: building the data pipelines and AI workloads that production AI agents depend on, and supporting the productionization of agentic systems by managing the underlying LLM infrastructure.
She holds a Master of Science in Big Data from Simon Fraser University, Vancouver, Canada.
Links
Area of Expertise
Rapid Deployment of LLMs into Production: Strategies and Insights
In the fast-paced domain of generative AI, the deployment of Large Language Models (LLMs) into production settings introduces a distinctive blend of challenges and opportunities. This presentation will detail our experience in incorporating LLMs into our product line within a challenging two-month period, a move motivated by the transformative potential of generative AI for enhancing our offerings. We navigated through various obstacles, such as constrained planning timelines, shifting requirements, the management of diverse stakeholder expectations, adaptation to emerging technologies, and the coordination of simultaneous workflows. These hurdles highlighted the pivotal role of data science and machine learning engineering teams in facilitating LLM integration, emphasizing the importance of security, testing, monitoring, and the pursuit of alternative solutions.
We will share the systematic approach we employed for identifying LLM use cases, validating their feasibility, engineering effective prompts, and crafting a comprehensive testing strategy. Additionally, we will introduce the LLM Service, a custom solution designed to ensure secure and efficient LLM access. This service underscores the significance of robust security protocols, the protection of customer data, the flexibility to switch LLM models to optimize performance for specific use cases, and the provision of redundancy in case of provider outages. Our discussion aims to illuminate how our expedited LLM deployment signifies the dawn of a new era in AI-driven product innovation.
RAG: From Prototype App to Production API
Developing a Retrieval-Augmented Generation (RAG) application requires addressing multiple complexities. The app must have reliable knowledge bases and robust data retrieval methods to query and filter extensive datasets. It also needs context awareness to generate accurate and coherent responses. To maintain optimal performance, it must also be scalable with quick response times even under high loads. Additionally, security and compliance are crucial when dealing with sensitive data and regulatory requirements.
Organizations can leverage internal prototyping and data science development tools to evaluate complex applications like RAG with real-time interactions and feedback mechanisms. These tools also enable capturing session histories, allowing data scientists to iterate quickly based on real-time data. This approach promotes stakeholder engagement and buy-in before involving engineering teams, providing other advantages like isolation from production, efficient use of data scientists' time, optimized engineering effort, and increased stakeholder confidence.
In this presentation, we'll discuss our journey with a RAG app, describing how we moved from a Streamlit-based prototype to a FastAPI production-ready API. We'll highlight the benefits of building a prototype and cover key steps in productionizing a RAG app. This includes defining essential RAG components, designing the architecture, selecting suitable models, identifying knowledge bases, setting up a vector store, handling chat history, implementing APIs, ensuring scalability, and maintaining compliance and security for the production environment.
Prototype AI Apps for Faster Innovation
Data scientists often struggle to share their work with stakeholders and get prompt feedback due to a lack of user-friendly platforms for real-time interaction with their models. Without these tools, the feedback process becomes slow and cumbersome, making it harder to iterate on ideas and improve their work.
At PagerDuty, we found that the absence of real-time interaction tools caused a ripple effect, slowing innovation and leading to delayed feedback loops, hindering model refinement. This also resulted in reduced collaboration among cross-functional teams, as stakeholders couldn't easily engage with data science projects. Ultimately, this leads to missed opportunities for innovation, especially in rapidly changing areas like machine learning and Generative AI. Without real-time tools, data scientists risk falling behind, compromising the organization's ability to remain competitive and adapt to technological changes.
A DS development platform like Streamlit can help solve these issues by enabling real-time interaction with AI Apps. It allows data scientists to share their work more easily, leading to faster feedback and improved collaboration, giving stakeholders confidence in the product before it's productionized.
This presentation will delve into the benefits that a DS development tool provides such as speed, collaboration, testing etc. which all contribute to quickly moving a project from ideation to production. Additionally, we will also walk through an example of a RAG GenAI application that started as a prototype but eventually evolved into a production feature.
Behind the Scenes: Powering PagerDuty's GenAI Features with LLMs
In the rapidly evolving landscape of incident management and response, PagerDuty stands at the forefront, integrating Generative AI (GenAI) features to enhance operational efficiency and user experience. This session delves into the intricate design and implementation of the API that power these GenAI features, focusing on the seamless integration of Large Language Models (LLMs) to automate and improve various aspects of the PagerDuty platform.
The core of our discussion revolves around the PagerDuty Copilot AI Assistant, which, through Slack integration, enables users to submit queries directly, thereby initiating the GenAI functionalities. At the heart of these operations is the Engine Core, which, upon receiving user queries, determines the appropriate task, fetches relevant data through the Data Fetcher from sources like the PagerDuty Operation Cloud and 3rd Party Services, and directs the request to the suitable backend service. This process is crucial for generating accurate, context-aware responses and actions.
Central to our discussion is the Data Science LLM API, which manages the interactions with various LLMs providers, such as OpenAI and AWS Bedrock. This API is pivotal in managing prompts, ensuring data security, and adhering to legal and security guidelines, thereby streamlining the implementation of GenAI features across the platform.
This session aims to provide a comprehensive overview of the backend processes and API designs that enable the GenAI features of PagerDuty, illustrating the platform's commitment to leveraging cutting-edge technology to enhance user experience and operational efficiency in incident management.
Suchita Venugopal
Senior Machine Learning Engineer, Pagerduty
Vancouver, Canada
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top