Understanding Communication Characteristics of Distributed Training
Name
3663408.3663409.pdf
Size
1.07 MB
Format
Adobe PDF
Checksum (MD5)
22023bd2cfa0fc4227b431e619b50536
Author(s)
Li, Wenxue
Liu, Xiangzhou
Li, Yuxuan
Jin, Yilun
Tian, Han
Zhong, Zhizhen
Liu, Guyue
Zhang, Ying
Chen, Kai
Date Issued
August 3, 2024
Publisher
ACM|The 8th Asia-Pacific Workshop on Networking
Citation
Wenxue Li, Xiangzhou Liu, Yuxuan Li, Yilun Jin, Han Tian, Zhizhen Zhong, Guyue Liu, Ying Zhang, and Kai Chen. 2024. Understanding Communication Characteristics of Distributed Training. In Proceedings of the 8th Asia-Pacific Workshop on Networking (APNet '24). Association for Computing Machinery, New York, NY, USA, 1–8.
Version
Final published version
Abstract
Communication is pivotal in distributed training and a thorough understanding of its characteristics is essential for future optimizations. However, prior works are limited, either focusing on customized optimizations or conducting incomplete explorations on communication characteristics. In this work, we systematically analyze the communication characteristics of distributed training, considering two key aspects of communication: pattern and overhead, and assessing a broad spectrum of determinant factors. In particular, we extensively investigate the features of communication patterns, such as predictability, and comprehensively evaluate the impact of various factors on communication overhead. Additionally, we develop and validate an analytical formulation to estimate communication overhead, providing a mathematical understanding of models with predictability.
Description
APNet 2024, August 03–04, 2024, Sydney, Australia
MIT Department
MIT Schwarzmann College of Computing
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3663408.3663409
