10 Steps to Transform RAG from POC to Production: The Hidden Complexity of RAG

In this talk, we will explore the journey of the Cisco DevNet engineering team as we transformed a simple Retrieval-Augmented Generation (RAG) proof of concept (POC) into a robust, production-ready application for developer.cisco.com. While creating a basic RAG demo is straightforward, scaling it to meet production standards involves navigating a myriad of challenges and complexities. This session will provide a step-by-step guide to the key aspects of this transformation, including vector indexing, model selection, prompt engineering, agent frameworks, security, user feedback, and continuous improvement.

Key Takeaways and Outline:

1. Introduction: Brief overview of RAG and its applications, highlighting the simplicity of creating a basic RAG demo.

2. Vector Indexing: Selecting the right vector database (Pinecone, Faiss, Milvus) and chunking strategies based on content type. Reference to VectorView benchmark.

3. Embedding Models: Choosing the right embedding model based on context size and balancing performance and cost.

4. Model Selection: Exploring self-hosting vs. API-based models with references to Artificial Analysis, The Fastest AI, and Hugging Face Open LLM Leaderboard.

5. Prompt Engineering: Crafting effective prompts, query rewriting, and reranking models.

6. Agent Frameworks: Comparing Langchain, Langgraph, and Autogen, and choosing the right framework for your needs.

7. Security and Guardrails: Adhering to the GenAI OWASP Top 10, implementing data privacy, access controls, and monitoring. Using Red Teaming with Promptfoo to simulate attacks and identify vulnerabilities, ensuring the application is robust and secure.

8. User Feedback and Trial Runs: Conducting trial runs, gathering user feedback, and benchmarking user trial queries. Using DeepEval for evaluating model performance, measuring accuracy, precision, recall, and F1 score, and integrating into

9. CI/CD pipelines for continuous monitoring.
Continuous Improvement: Setting up a dashboard to track, analyze, and improve patterns, making data-driven decisions to enhance the user experience.

10. Deployment: Automating the deployment process with CI/CD pipelines, real-time monitoring and logging, and rollback strategies for safe deployments.

Conclude with Real-World Application: Case study of developer.Cisco.com, discussing challenges faced and solutions implemented.

https://developerweek2025.sched.com/event/1tFJk/open-session-10-steps-to-transform-rag-from-poc-to-production-the-hidden-complexity-of-rag

Building a Developer Oasis: Designing and Orchestrating Cloud-Based Developer Pods for Large Develop

In the session we will delve into the journey of constructing a scalable and efficient cloud developer environment, leveraging the power of Kubernetes, to cater to over 10,000+ developers and learners. This session aims to provide attendees with valuable insights into the challenges faced, strategies employed, and best practices adopted throughout the implementation process.

We will commence by highlighting the significance of a consistent and controlled workspace for developers, particularly when accommodating a substantial number of individuals. We will then explore the unique challenges and considerations associated with designing such an environment, encompassing aspects like infrastructure scaling, security and access control, and resource management, all within the context of Kubernetes.

The presentation will encompass the design elements of the developer pod infrastructure, where Kubernetes plays a crucial role. We will examine how Kubernetes enables container orchestration, ensuring the seamless deployment, scaling, and management of developer pods. We will explore the benefits of leveraging Kubernetes for workload isolation, resource optimization, and improved developer productivity.

Furthermore, we will delve into the configuration and customization aspects, focusing on utilizing Kubernetes capabilities to pre-install dependencies, facilitate integration with Cisco technologies and APIs, and empower developers to tailor their pods to suit specific project requirements.

By attending this session, participants will acquire valuable insights into building and managing a cloud development environment utilizing Kubernetes for a vast developer community. Armed with this knowledge and the shared best practices, they will be well-equipped to implement scalable and efficient Kubernetes-based environments within their own developer organization.

CloudX Developer Week: https://sched.co/1QCeT

Elevating API Quality: Strategies for Accelerating API Excellence in Large Organizations

Cisco has embraced an API-First strategy to drive innovation, collaboration, and efficiency within its extensive product ecosystem. Join us in this compelling talk as we explore the unique challenges organizations face in maintaining high-quality APIs at scale and discover effective strategies to implement a comprehensive organization-wide API Quality program.

To overcome these challenges, organizations must adopt an automation and data-driven approach that provides actionable insights into API quality. We have developed powerful tooling that automatically analyzes and scores OpenAPI docs against API guidelines, detects backward compatibility breaking changes, and publishes a centralized API Quality dashboard.

Drawing upon our experience, we will share our journey in developing and open-sourcing "API Insights." This invaluable tool empowers organizations to monitor, analyze, and improve API quality based on our accumulated knowledge and expertise.

This session will provide practical strategies and actionable takeaways to accelerate API quality initiatives in your organization. Leveraging API Insights’ VS Code extension, dashboard, and integration with your CI/CD workflow, you can elevate your API development and maintenance practices to new heights, resulting in improved developer experiences, enhanced customer satisfaction, and increased business success.

API World 2023: https://apiworld2023.sched.com/event/1UBNK?iframe=no

AI in Action: 10 Transformative Use Cases in API Documentation and Developer Portals

Artificial Intelligence has begun to reshape the landscape of API documentation and developer portals, though its integration is still in the early stages. This talk will present an in-depth look at the top ten use cases where AI proves its game-changing potential.

We will explore topics such as personalized recommendation engines, AI-enabled semantic search/summary generation, chatbots, content tagging, sentiment analysis, video searchability, code generation, API quality improvement, and AI-powered authoring assistants.

Moreover, we will address the current challenges encountered in these areas, as well as the ongoing efforts to overcome them. We also intend to share current knowledge and insights about the implementation and impact of AI in this domain. Furthermore, the session will provide an overview of a potential architecture that integrates these AI applications for optimized functionality and improved user experience.

Join us as we explore the emerging landscape of AI applications in API documentation and developer portals, the challenges we face, and the exciting possibilities for the future.
https://apithedocs.org/ai-2024/neelesh-pateriya

12 Facets of the OpenAPI Specification (APIWorld2022)

We'll introduce how Cisco Engineering leverages OAS to drive API quality and state-of-the-art developer experience. We'll then describe OpenAPI best practices, tools and processes built internally and opensourced, as well as the benefits for Cisco partners and customers. Join this session to hear from the best practices and lessons learnt when standardizing on OAS for organizations with a massive internal and external facing APIs portfolio.
https://apiworld.co/