Congestion Control in Machine Learning Clusters
Name
rajasekaran-rsudhir-sm-eecs-2024-thesis.pdf
Description
Thesis PDF
Size
2.08 MB
Format
Adobe PDF
Checksum (MD5)
194a8f4f29437daa8795ca79f09e3142
Author(s)
Rajasekaran, Sudarsanan
Advisor(s)
Ghobadi, Manya
Date Issued
May 2024
Publisher
Massachusetts Institute of Technology
Abstract
This paper argues that fair-sharing, the holy grail of congestion control algorithms for decades, is not necessarily a desirable property in Machine Learning (ML) training clusters. We demonstrate that for a specific combination of jobs, introducing unfairness improves the training time for all competing jobs. We call this specific combination of jobs compatible and define the compatibility criterion using a novel geometric abstraction. Our abstraction rolls time around a circle and rotates the communication phases of jobs to identify fully compatible jobs. Using this abstraction, we demonstrate up to 1.3× improvement in the average training iteration time of popular ML models. We advocate that resource management algorithms should take job compatibility on network links into account. We then propose three directions to ameliorate the impact of network congestion in ML training clusters: (i) an adaptively unfair congestion control scheme, (ii) priority queues on switches, and (iii) precise flow scheduling.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
