When we need to scale a product, automation is almost always the right answer, and it’s easy to reach for Artificial Intelligence (AI) because it’s so broadly applicable. However, a general-purpose tool like AI often underperforms when compared to more specialized tools. This talk is a case study about a product team that was scaling their product (scoring assignments against a rubric) with AI (using GenAI, NLP, and ML) and not seeing the results they expected. In this talk we will discuss why and the special-purpose techniques we used in place of AI.

By the end of the talk, the audience will have been introduced to two models for AI-based natural language and code understanding. We will establish some heuristics for deciding when AI is necessary or when a more specialized technique is likely be more desirable. These discussions don’t require a fluency in AI itself, GenAI, ML, NLP, or any of the other technologies we used. The important part is a desire to find the right technology for a given problem.