Understanding Communication Characteristics of Distributed Training
Author(s)
Li, Wenxue; Liu, Xiangzhou; Li, Yuxuan; Jin, Yilun; Tian, Han; Zhong, Zhizhen; Liu, Guyue; Zhang, Ying; Chen, Kai; ... Show more Show less
Download3663408.3663409.pdf (1.066Mb)
Publisher Policy
Publisher Policy
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Terms of use
Metadata
Show full item recordAbstract
Communication is pivotal in distributed training and a thorough understanding of its characteristics is essential for future optimizations. However, prior works are limited, either focusing on customized optimizations or conducting incomplete explorations on communication characteristics. In this work, we systematically analyze the communication characteristics of distributed training, considering two key aspects of communication: pattern and overhead, and assessing a broad spectrum of determinant factors. In particular, we extensively investigate the features of communication patterns, such as predictability, and comprehensively evaluate the impact of various factors on communication overhead. Additionally, we develop and validate an analytical formulation to estimate communication overhead, providing a mathematical understanding of models with predictability.
Description
APNet 2024, August 03–04, 2024, Sydney, Australia
Date issued
2024-08-03Department
MIT Schwarzmann College of ComputingPublisher
ACM|The 8th Asia-Pacific Workshop on Networking
Citation
Wenxue Li, Xiangzhou Liu, Yuxuan Li, Yilun Jin, Han Tian, Zhizhen Zhong, Guyue Liu, Ying Zhang, and Kai Chen. 2024. Understanding Communication Characteristics of Distributed Training. In Proceedings of the 8th Asia-Pacific Workshop on Networking (APNet '24). Association for Computing Machinery, New York, NY, USA, 1–8.
Version: Final published version
ISBN
979-8-4007-1758-1