Carbon- and Precedence-Aware Scheduling for Data Processing Clusters
Name
3718958.3750478.pdf
Size
1.06 MB
Format
Adobe PDF
Checksum (MD5)
2b5cbe300d1faac3d1012d512af1d2fc
Author(s)
Lechowicz, Adam
Shenoy, Rohan
Bashir, Noman
Hajiesmaili, Mohammad
Wierman, Adam
Delimitrou, Christina
Date Issued
August 27, 2025
Publisher
ACM|ACM SIGCOMM 2025 Conference
Citation
Adam Lechowicz, Rohan Shenoy, Noman Bashir, Mohammad Hajiesmaili, Adam Wierman, and Christina Delimitrou. 2025. Carbon- and Precedence-Aware Scheduling for Data Processing Clusters. In Proceedings of the ACM SIGCOMM 2025 Conference (SIGCOMM '25). Association for Computing Machinery, New York, NY, USA, 1241–1244.
Version
Final published version
Abstract
As large-scale data processing workloads continue to grow, their carbon footprint raises concerns. Prior research on carbon-aware schedulers has focused on shifting computation to align with the availability of low-carbon energy, but these approaches assume that each task can be executed independently. In contrast, data processing jobs have precedence constraints that complicate decisions, since delaying an upstream "bottleneck" task to a low-carbon period also blocks downstream tasks, impacting makespan. In this paper, we show that carbon-aware scheduling for data processing benefits from knowledge of both time-varying carbon and precedence constraints. Our main contribution is PCAPS, a carbon-aware scheduler that builds on state-of-the-art scoring or probability-based techniques - in doing so, it explicitly relates the structural importance of each task against the time-varying characteristics of carbon intensity. To illustrate gains due to fine-grained task-level scheduling, we also study CAP, a wrapper for any carbon-agnostic scheduler that generalizes the provisioning ideas of PCAPS. Both techniques allow a user-configurable priority between carbon and makespan, and we give basic analytic results to relate the trade-off between these objectives. Our prototype on a 100-node Kubernetes cluster shows that a moderate configuration of PCAPS reduces carbon footprint by up to 32.9% without significantly impacting total efficiency.
Description
SIGCOMM ’25, Coimbra, Portugal
MIT Department
Massachusetts Institute of Technology. School of Engineering
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3718958.3750478
