Machine Learning has been advancing rapidly, but only a few contributors are focusing on the infrastructure and scaling challenges that come with it. When you have thousands of model versions, each written in any mix of frameworks (Python/R/Java/Ruby, PyTorch/SciKit/Caffe/Tensorflow etc), how do you efficiently deploy them as elastic, scalable, secure APIs with 10ms of latency and GPU access?

We’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework. We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them.

In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI”: a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.