Precision Machine Learning
Name
entropy-25-00175-v3.pdf
Size
3.38 MB
Format
Adobe PDF
Checksum (MD5)
0ca4e8ba49ad8d4e5b02b6320220af38
Author(s)
Michaud, Eric J.
Liu, Ziming
Tegmark, Max
Date Issued
January 15, 2023
Publisher
Multidisciplinary Digital Publishing Institute
Citation
Entropy 25 (1): 175 (2023)
Version
Final published version
Abstract
We explore unique considerations involved in fitting machine learning (ML) models to data with very high precision, as is often required for science applications. We empirically compare various function approximation methods and study how they scale with increasing parameters and data. We find that neural networks (NNs) can often outperform classical approximation methods on high-dimensional examples, by (we hypothesize) auto-discovering and exploiting modular structures therein. However, neural networks trained with common optimizers are less powerful for low-dimensional cases, which motivates us to study the unique properties of neural network loss landscapes and the corresponding optimization challenges that arise in the high precision regime. To address the optimization issue in low dimensions, we develop training tricks which enable us to train neural networks to extremely low loss, close to the limits allowed by numerical precision.
MIT Department
Massachusetts Institute of Technology. Department of Physics
Center for Brains, Minds, and Machines
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.3390/e25010175
