The Impact of Internal Variability on Benchmarking Deep Learning Climate Emulators
Name
J Adv Model Earth Syst - 2025 - Lütjens - The Impact of Internal Variability on Benchmarking Deep Learning Climate.pdf
Description
Published version
Size
7.48 MB
Format
Adobe PDF
Checksum (MD5)
f0144604e3284c1393924608a4eb52d7
Author(s)
Lütjens, Björn
Ferrari, Raffaele
Watson‐Parris, Duncan
Selin, Noelle E
Date Issued
August 26, 2025
Journal
Journal of Advances in Modeling Earth Systems
Publisher
Wiley
Citation
Lütjens, B., Ferrari, R., Watson-Parris, D., & Selin, N. E. (2025). The impact of internal variability on benchmarking deep learning climate emulators. Journal of Advances in Modeling Earth Systems, 17, e2024MS004619.
Version
Final published version
Abstract
Full-complexity Earth system models (ESMs) are computationally very expensive, limiting their use in exploring the climate outcomes of multiple emission pathways. More efficient emulators that approximate ESMs can directly map emissions onto climate outcomes, and benchmarks are being used to evaluate their accuracy on standardized tasks and data sets. We investigate a popular benchmark in data-driven climate emulation, ClimateBench, on which deep learning-based emulators are currently achieving the best performance. We compare these deep learning emulators with a linear regression-based emulator, akin to pattern scaling, and show that it outperforms the incumbent 100M-parameter deep learning foundation model, ClimaX, on 3 out of 4 regionally resolved climate variables, notably surface temperature and precipitation. While emulating surface temperature is expected to be predominantly linear, this result is surprising for emulating precipitation. Precipitation is a much more noisy variable, and we show that deep learning emulators can overfit to internal variability noise at low frequencies, degrading their performance in comparison to a linear emulator. We address the issue of overfitting by increasing the number of climate simulations per emission pathway (from 3 to 50) and updating the benchmark targets with the respective ensemble averages from the MPI-ESM1.2-LR model. Using the new targets, we show that linear pattern scaling continues to be more accurate on temperature, but can be outperformed by a deep learning-based technique for emulating precipitation. We publish our code and data at https://github.com/blutjens/climate-emulator.
MIT Department
Massachusetts Institute of Technology. Department of Earth, Atmospheric, and Planetary Sciences
MIT Institute for Data, Systems, and Society
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1029/2024MS004619
