Sho Tanaka
Developer Advocate
Tokyo, Japan
Actions
Lead Developer Advocate at Snowflake, focused on AI/ML and data engineering. He previously worked at Google (gTech) delivering ML/Data solutions across Japan, APAC and global. He is a Google Developer Expert (AI/ML) and a co-founder of the MLOps community in Japan, where he has organized monthly events for over five years for a growing network of 6,000+ members. He enjoys turning messy real-world ML projects into reproducible, production-minded architectures.
Area of Expertise
Topics
When MCP Tools Fail: Reliable ML Inference for MCP Agents
The Model Context Protocol (MCP) is emerging as a standard for connecting AI agents with external tools. While LLM-based tools are flexible, their non-deterministic behavior makes it difficult to meet requirements for stability, repeatability, and auditability in production systems. In such cases, traditional machine learning models provide more predictable inference and are often preferred for system-critical decisions.
However, ML-powered tools are not free from challenges. Latency, failures, and integration issues can propagate through MCP systems and impact agent behavior, leading to inconsistent decisions, repeated retries, or silent failures.
This session explores what happens when MCP tools fail and how to design systems that remain reliable under such conditions. We present a reproducible demo of failure scenarios, including latency spikes and unstable predictions, and show their impact on agent behavior. We then introduce practical design patterns—such as timeout and retry strategies and observability—to build reliable MCP-based systems.
Attendees will gain concrete techniques to design trustworthy AI systems that remain stable in real-world environments.
Building Reliable AI Agents: An Open Source Approach to Evaluation and Observability
Building AI agents is easier than ever with open source tools, but ensuring their reliability in production remains a major challenge. Unlike traditional software, AI agents are non-deterministic, making simple pass/fail testing insufficient.
This talk introduces a practical approach to evaluation and observability for AI agents, combining open source tools such as TruLens with agent architectures inspired by AgentGPT.
We will demonstrate how to instrument agent workflows, capture execution traces, and implement evaluation metrics such as faithfulness, tool selection accuracy, and answer relevancy. Attendees will also see how to visualize agent behavior and identify failure points across retrieval, reasoning, and generation layers using a lightweight dashboard.
Finally, we show how to build a feedback loop to iteratively improve agent performance, and share a reference implementation (GitHub) that can be reused with different agent frameworks.
Model Evaluation for Custom Datasets with Open Models: Multi-Model Comparison with Streamlit
Open-source embedding models often claim multilingual support, but their Japanese language performance varies significantly from English benchmarks. This talk presents a evaluation idea for specific-lang assessment of open-source embedding models with custom data.
I demonstrate a Streamlit-based platform comparing Japanese-specific models (Ruri series), multilingual alternatives (multilingual-E5, BGE-M3), and general-purpose models across real-world Japanese tasks including semantic search, etc.
All code methodologies will be released as open source, enabling developers worldwide to learn from our approach and adapt the framework for their own languages and cultural contexts using entirely open-source tools.
MCP Dev Summit Seoul 2026 Sessionize Event
Open Source Summit Korea 2026 Sessionize Event
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top