Priyan Pattnayak
Senior Principal Scientist - Oracle Cloud AI
Seattle, Washington, United States
Actions
Priyaranjan (Priyan) Pattnayak is a Senior Principal Data Scientist at Oracle Cloud working on agentic and conversational AI systems for enterprise support. He builds evaluation infrastructure for multi-turn conversational experiences, including customer simulation frameworks that measure completion and correctness at scale. His work focuses on making AI assistants reliable in production through structured evaluation, failure attribution, and system-level design. He has published/filed over 30 papers and patents in top tier forums and is an active researcher in the NLP community.
Area of Expertise
Topics
IndicSafe: Evaluating Multilingual LLM Safety in South Asia
As large language models (LLMs) are deployed in multilingual settings, their safety behavior in culturally diverse, low-resource languages remains poorly understood. We present the first systematic evaluation of LLM safety across 12 Indic languages, spoken by over 1.2 billion people but underrepresented in LLM training data. Using a dataset of 6,000 culturally grounded prompts spanning caste, religion, gender, health, and politics, we assess 10 leading LLMs on translated variants of the prompt.
Our analysis reveals significant safety drift: cross-language agreement is just 12.8\%, and \texttt{SAFE} rate variance exceeds 17\% across languages. Some models over-refuse benign prompts in low-resource scripts, overflag politically sensitive topics, while others fail to flag unsafe generations. We quantify these failures using prompt-level entropy, category bias scores, and multilingual consistency indices.
Our findings highlight critical safety generalization gaps in multilingual LLMs and show that safety alignment does not transfer evenly across languages. We release \textsc{IndicSafe}, the first benchmark to enable culturally informed safety evaluation for Indic deployments, and advocate for language-aware alignment strategies grounded in regional harms.
Agentic Support on OCI: Tool-Using Assistants and Evaluation
Agents can call tools, retrieve context, and execute multi-step plans, but they introduce new failure modes (wrong tool, wrong plan, policy drift). This talk walks through a practical approach to building tool-using support assistants on OCI and, more importantly, evaluating them stage-by-stage. You’ll learn how to detect regressions, attribute failures to retrieval vs planning vs response, and deploy safely with measurable quality thresholds.
Beyond the AI Demo: How to Ship LLM Features That Actually Work in Production
Everyone can build an AI demo. Shipping reliable, safe, measurable LLM features is the hard part. This talk shares a practical framework for turning GenAI prototypes into production systems: defining success metrics, designing evals, reducing hallucinations, building feedback loops, and keeping cost/latency under control. Attendees will leave with a repeatable LLM product checklist and patterns they can apply immediately, whether they’re building copilots, search, support automation, or internal knowledge tools.
Chaos Testing for Chatbots: Simulating Customers to Evaluate AI Agents
Most conversational AI demos look great in single-turn prompts. But real customers don’t behave like prompts: they interrupt, change goals mid-way, provide incomplete information, and ask follow-ups that force the system to stay consistent across multiple steps.
In this session, I’ll share how we built an AI Simulator to evaluate multi-turn conversational systems in a realistic way. Instead of testing a chatbot with isolated prompts, we simulate complete customer journeys, troubleshooting flows, account issues, configuration tasks and automatically measure task completion, correctness, and recovery behavior when the agent makes mistakes.
You’ll learn how multi-turn simulation exposes failure modes that traditional evaluation misses (wrong tool usage, premature answers, policy violations, drift across turns, and “confidently wrong” resolution). We’ll cover the design of customer personas, scenario templates, success criteria, and how to turn simulation results into a production-grade metric suite that enables regression testing and reliable iteration.
If you're building agents, RAG assistants, or support chatbots, this talk will show you how to evaluate them like real systems, not like demos.
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top