With the development of AI and Quant, more and more elastic jobs were introduced into those areas with performance and security requirements, e.g. matrix multiplication, Monte Carlo. As an elastic job, there may be thousands of tasks which did not depends on each other, e g. matrix multiplication; so, any task can re-run/re-try at any time; the tasks may share dataset by cache or distributed filesystem. Considering the number of tasks, the performance, throughput and resource utilization is important; and security is also important for a multi-tenant platform.

For those scenarios, a distributed engine, named Flame, was introduced. It includes several features and enhancement, e.g. fair-share, preemption, pull-model, session/tasks, for performance, throughput.

This session will present the architect and features of Flame; it'll also demonstrate the improvement by matrix multiplication, Monte Carlo and so on.