Enabling Efficient ML Inference in SigmaOS with Model-Aware Scheduling
Name
liu-katieliu-meng-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
2.06 MB
Format
Adobe PDF
Checksum (MD5)
d4711d510e1eb4b2e7443a4e11381b56
Author(s)
Liu, Katie
Advisor(s)
Szekely, Ariel
Kaashoek, M. Frans
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
Machine learning inference in multi-tenant cloud environments leads to significant challenges when it comes to minimizing latency and resource contention, especially as models grow in size and complexity. This thesis addresses the cold start overhead and scheduling inefficiencies of multi-tenant ML serving by integrating the RayServe distributed model-serving framework into σOS, a cloud operating system that unifies container and serverless paradigms. The thesis also proposes two model-aware schedulers within σOS that intelligently routes inference requests to reduce the number of cold starts: Model Colocation, which prioritizes placing requests on machines where the required model is already loaded, and Centralized Model Registry, which tracks globally available models to inform scheduling decisions. These policies proactively reduce model load times by reusing cached models. Experimental results on language translation workloads in an 8-node cluster show that these schedulers achieve a ≈ 50% reduction in average inference latency and eliminates roughly 4–5 cold starts per workload, compared to σOS’s default scheduler. Through this model-aware approach to scheduling, our work enables more efficient, scalable, and low-latency ML inference serving in multi-tenant cloud settings.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
