LightSpeed: A Framework to Profile and Evaluate Inference Accelerators at Scale

Abstract

The massive growth of machine learning-based applications, and the end of Moore’s law, created a pressing need to build highly efficient computing platforms from the ground up. Consequently, researchers and practitioners have been developing highly innovative cutting-edge architectures to meet today’s exponentially increasing demands for machine learning services.

However, evaluating the performance gains of newly developed machine learning systems at scale is extremely challenging. Existing evaluation platforms are often specialized to a specific hardware target, such as GPUs, making them less amenable to novel designs. Moreover, evaluating the performance of a newly designed system at scale requires careful consideration of workload and traffic patterns.

To address the above challenges, I introduce LightSpeed, a framework to profile and evaluate inference accelerators at scale. LightSpeed is an event-based simulator that enables users to compare the performance of their system to best-in-class accelerators at scale. LightSpeed profiles the computation and communication requirements of real-world deep neural networks through accurate measurements on hardware. It then simulates the service time of inference requests under a variety of accelerators and scheduling algorithms.