Open-source embedding models often claim multilingual support, but their Japanese language performance varies significantly from English benchmarks. This talk presents a evaluation idea for specific-lang assessment of open-source embedding models with custom data.

I demonstrate a Streamlit-based platform comparing Japanese-specific models (Ruri series), multilingual alternatives (multilingual-E5, BGE-M3), and general-purpose models across real-world Japanese tasks including semantic search, etc.

All code methodologies will be released as open source, enabling developers worldwide to learn from our approach and adapt the framework for their own languages and cultural contexts using entirely open-source tools.