Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training
Name
3767295.3769369.pdf
Size
1.9 MB
Format
Adobe PDF
Checksum (MD5)
fbd8a4dcc9fd1070228858ce61aa3687
Author(s)
Chen, Chang
Chen, Tiancheng
Duan, Jiangfei
Zhu, Qianchao
Wang, Zerui
Hu, Qinghao
Sun, Peng
Li, Xiuhong
Yang, Chao
Hoefler, Torsten
Date Issued
April 26, 2026
Publisher
Association for Computing Machinery
Citation
Chang Chen, Tiancheng Chen, Jiangfei Duan, Qianchao Zhu, Zerui Wang, Qinghao Hu, Peng Sun, Xiuhong Li, Chao Yang, and Torsten Hoefler. 2026. Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training. In Proceedings of the 21st European Conference on Computer Systems (EUROSYS '26). Association for Computing Machinery, New York, NY, USA, 1879–1893.
Version
Final published version
Abstract
Training large language models (LLMs) with increasingly long and varying sequence lengths introduces severe load imbalance challenges in large-scale data-parallel training. Recent frameworks attempt to mitigate these issues through data reorganization or hybrid parallel strategies. However, they often overlook how computational and communication costs scale with sequence length, resulting in suboptimal performance. We identify three critical challenges: (1) varying computation-to-communication ratios across sequences of different lengths in distributed attention, (2) mismatch between static NIC-GPU affinity and dynamic parallel workloads, and (3) distinct optimal partitioning strategies required for quadratic attention versus linear components.
To address these challenges, we present Zeppelin, a novel training system that integrates three key techniques: (1) a hierarchical sequence partitioning method for the attention module that reduces communication overhead and balances computation, supported by an efficient attention engine that applies divergent parallel strategies; (2) a routing layer that orchestrates inter-node transfers to fully utilize NIC bandwidth; and (3) a remapping layer that transforms sequence layouts between attention and linear modules, ensuring high computational efficiency across both. Comprehensive evaluations across diverse configurations show that Zeppelin delivers an average 2.80× speedup over state-of-the-art methods.
Description
EUROSYS ’26, Edinburgh, Scotland Uk
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3767295.3769369
