Sangeeth Kumar
I'm an Engineering Manager for AI Transformation at Kissflow
Chennai, India
Actions
I'm an Engineering Manager for AI Transformation at Kissflow. I spent the first decade of my career in cloud infrastructure — orchestration across AWS and OpenStack, provisioning workflows, identity and access control — which turns
out to be decent preparation for what I do now, because both jobs are mostly about state you can't see.
These days I work on the context and evaluation layer behind an internal AI assistant: the thing engineers ask when they need to know which service throws a given error code across a 7,000-file Python monorepo.
Most of my time goes to the unglamorous half of that problem. Not getting the model to answer, but working out whether the answer was any good. That's meant a lot of golden sets, a few LLM judges that turned out to be the broken part,
and 42 numbered experiments of which six actually worked.
I write about RAG evaluation and context engineering on LinkedIn. Based in Chennai.
Area of Expertise
Topics
Build & Break a Context Layer — Hands-On Evaluation for RAG and Agents
I spent three months making our RAG system better. Then I actually measured it and found I'd mostly been shipping noise.
Some context: we'd built a knowledge layer over a large Python monorepo, the kind of thing that answers "which service throws this error code" for engineers. Retrieval metrics looked fine. Dashboard was green. Users kept telling us the answers were wrong.
It wasn't the model or the chunking. Our evaluation just couldn't tell a real improvement apart from a coin flip — run it twice on the same code, get two different scores. We'd promoted a change on a 3-point gain that turned out to sit inside the error bar.
So in this session I'll build a small context layer live, then break it in front of you. Bad eval, bad benchmark, a judge that scores the wrong answer higher than the right one, and a document I poison halfway through that the pipeline never notices. Some of this is embarrassing — a stronger model beat our benchmark by 16 points and got measurably worse on real user traffic.
Repo goes out at the end so you can run all of it yourself.
Takeaways
• Why your eval probably can't detect the improvement you just shipped
• A cheap way to get error bars on it without tripling what you spend on evals
• The boring governance work that moved our numbers more than any amount of
retrieval tuning did
Format Session — 40 minutes (incl. Q&A)
Tags Gen AI, LLM · Agents, RAG · Data
TechNexus Community Bengaluru CFS 2026 User group Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top