Resilient LLM - Why your LLM calls fail in production and how to make them resilient

This talk introduces a personal Open Source project that has been receiving a good reviews and shares the technical learning with the community.

When I deployed my AI agents to production, they started facing failures, some predictable (e.g. hitting LLM provider's rate limits), some unpredictable (Anthropic's overload error, network issues, CPU/memory spikes leading to server crash, etc.). Some of these issues were easy to deal with e.g. a simple exponential backoff and retry strategy. But it was not good enough to put it out there on production. I could have put a rate limit gateway in front of my app server but that wouldn't have enough user/app context/control to recover from these failures and leave the gap for unpredictable errors. Also it would have been an extra chore and expense to manage that gateway. So for the multiple agentic apps that I was creating, the LLM calls had to be more resilient, and the solution to deal with most of these failures had to be in the app itself.

Existing libraries such as Vercel AI added even more unpredictability e.g. AI_UnsupportedModelVersionError. So I found myself writing duplicate code to make my LLM apps more resilient. This motivated me to write this Open Source library: ResilientLLM - a resilient, unified LLM interface featuring circuit breaker, token bucket rate limiting, caching, and adaptive retry with dynamic backoff support.

In simpler words, it is a class that unifies API calls to any LLM via a common interface, the usage looks as following - unifiedAndResilientLLM.chat(conversationHistory, llmAndResilienceOptions). It frees up my time from worrying about the critical failures that LLM apps and AI agents face in production.

This minimalist Open Source library aims to solve the same challenges for you by providing a resilient layer that intelligently manages failures and rate limits, enabling you (developers) to integrate LLMs confidently and effortlessly at scale.

Key Takeaways
Understand common culprits why LLM apps fail on production e.g. unstable network, rate limits, unpredictable overload, etc.

Your first steps towards resilience - circuit breakers, token bucket rate limiting, graceful timeout and failures, adaptive retries

Practical and effective implementation of resilience patterns without moving focus away from the AI agent development

Excel developer experience like a pro Open Source maintainer

A bad Developer Experience leads to losing users to competing product, low team morale, and increased support load. With examples of good vs bad developer experience of some Open Source projects, you'll walk away how you can improve developer experience to improve adoption of your developer tool. The author has been uncovering such insights for more than 7 years now for his newsletter. This will be a compilation of most repeated examples of bad experience and the most impactful examples of good experience.

Autonomous AI agent for Penetration Testing - Current Status and Future

With the increasing Large Language Model's performance in reasoning, the possibility of an LLM-powered agent to autonomously perform pentesting is becoming a viable reality to some extent.
How well can an AI agent perform pentesting autonomously starting from scanning to finally drafting the Vulnerability Assessment and Penetration Testing report (VAPT)?
Pradeep will demo a proof of concept of such AI Agent that autonomously plans the pentesting steps, performs scanning, analyzes scanning result, enumerates, exploits vulnerability, and finally drafts a VAPT. Pradeep will summarise the effectiveness, challenges, and the lessons learned for future scope of development of such AI Cyber Security agent.

Private Linux Powered LLM Box As Second Brain

What if you can replace your notebook, laptop, phone, and everything else you use for journaling. Instead use a portable local LLM-powered hardware that will remember your ideas, answer your questions from your own brain from the past, take appropriate actions, etc. This session will cover how I built it for my home office using Open Source LLM models.

Building private LLM-powered second brain on Intel CPU

What if you can use the power of LLM and programming to remember your ideas, answer your questions from your own "brain from the past", take appropriate actions autonomously just in time! I experienced a massive transformation with such system. But doing this well on a consumer-grade hardware is not easy. In this talk, I will present different experiments I did to make it work on my home computer running on Linux and Intel chip. I will share what worked the best and how can you do the same completely offline, without using any cloud computing.

Don't blame the AI - Close the accountability gap with Agent Human Interaction protocol

AI agents are making real decisions in real workflows. When they get it wrong, and they will, who is accountable?

This talk introduces Agent-Human Interaction protocol, a practical framework to standardise how AI agents and humans collaborate and share accountability.

We will explore:
How the effective use of AI agents requires humans and AI working together as a team.
How mistakes can be avoided by enabling an effective Agent Human interaction and the ability to track the accountability of AI actions to the responsible human.
How an effective Agent Human interaction must acknowledge the human factors (e.g. decision fatigue) in usability and security as NIST recommends.

With examples, we will discuss how Agent Human Interaction protocol can standardize the smooth and transparent interactions between agents and human stakeholders.

This talk opens up a dialogue to collectively drive accountability, clarity, and trust in agent–human collaboration.

Responsible and effective use of AI for technical documentation

Creating and maintaining tech docs has been a challenge for developers. Naturally, it led to explore Gen AI to solve this boring and often complex task. Only to end up more burned out and making it even harder and slower for potential users to use their product.

In this talk, Pradeep shares his experiences doing the same as a Developer Relations Specialist. He shares his initial failures and the insights that led to dramatic improvements in tech docs. By the end of the talk, you'll learn the process which helped protecting the human aspect of tech docs while complimenting the weaknesses of humans maintaining the tech docs using LLM agents.

The first part of the talk will cover how to think about the documentation to avoid creating convoluting tech docs structure. Primarily organizing docs into four categories - Tutorial, Reference, How To Guides, Explanation.

The second part will cover the responsible automation using AI.

Simplifying JavaScript code for API integrations at scale

With the increasing dependency on external APIs and their complex data transformation requirements, the JavaScript code becomes hard to manage and scale. We will discuss these challenges. We will evaluate JSONata and custom JSON templating language/engine as a solution. After this talk, you'll be able to manage the API integration complexity in JavaScript or TypeScript, making the API data transformation code maintainable and scalable

The search journey from Apache Lucene to Open Search

Year 2010, Pradeep was wrangling with many GBs of xml files containing the data of Computer Science research authors. The goal was to build search on top of this data providing insights into their data. Apache Lucene and Solr were the leading search software space. Forget about the semantic search, the keyword search was a distant dream for a young developer like Pradeep. Fast forward to 2025, the search software ecosystem has evolved to the level where a young inexperienced developer can easily build multimodal AI-powered search within few days. In this session, Pradeep shares insights about how search systems, algorithms, and the entire search software ecosystem has evolved. We will also cover what are the challenges that are yet to be addressed, and how you and projects like Open Search play an important role in the future search journey.

Can adaptive personalized practice using AI replace 1:1 tutoring

Practice is critical for learning. 1:1 tutoring proves to be effective because the tutor can create personalized practice and keep adapting to the student's progress. Can AI do the same? Can AI do that for STEM? This discussion between math tutor and an AI developer will bring some eye opening perspectives to the audience.