Solving large optimization problems is challenging, yet when parallelization is possible, serverless computing can drastically reduce the elapsed time.

In this talk, we consider a real case study: creating power market scenarios by finding the optimal way of using power storage facilities.

This results in a convex optimization problem with hundreds of thousands of variables. Despite parallelization, a conventional solution requires around two hours on a 16-core virtual machine.

We discuss how different serverless approaches can be adopted to reduce the elapsed time and demonstrate how cloud functions help cut it by 96% while keeping costs as low as 3€ per run. Additionally, we show how to connect the optimization module to the rest of the data processing pipeline, implemented in AirFlow.

By the end of this talk, you will have a better understanding of the capabilities of serverless for high-performance computing.

First delivered at PyCon Italy, 24-28 May 2023, Florence, Italy
Recorded session at https://youtu.be/JAOK9Zut_R4