Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. MIT Open Access Articles
  3. MIT Open Access Articles
  4. Neural scaling of deep chemical models

Neural scaling of deep chemical models

Thumbnail Image
Download
Name

s42256-023-00740-3.pdf

Description
Published version
Size

1.67 MB

Format

Adobe PDF

Checksum (MD5)

9f91beefc55435f060cb44904eea38f4

sword-2025-02-11T20:58:28.original.xml (130 B)
Original SWORD entry document
Author(s)
Frey, Nathan C
•
Soklaski, Ryan
•
Axelrod, Simon
•
Samsi, Siddharth
•
Gómez-Bombarelli, Rafael
•
Coley, Connor W
•
Gadepally, Vijay
Date Issued
2023
Journal
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Citation
Frey, N.C., Soklaski, R., Axelrod, S. et al. Neural scaling of deep chemical models. Nat Mach Intell 5, 1297–1305 (2023).
Version
Final published version
Abstract
Massive scale, in terms of both data availability and computation, enables important breakthroughs in key application areas of deep learning such as natural language processing and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of physical priors in scientific domains makes the strategies and benefits of scaling uncertain. Here we investigate neural-scaling behaviour in large chemical models by varying model and dataset sizes over many orders of magnitude, studying models with over one billion parameters, pre-trained on datasets of up to ten million datapoints. We consider large language models for generative chemistry and graph neural networks for machine-learned interatomic potentials. We investigate the interplay between physical priors and scale and discover empirical neural-scaling relations for language models in chemistry with a scaling exponent of 0.17 for the largest dataset size considered, and a scaling exponent of 0.26 for equivariant graph neural network interatomic potentials.
MIT Department
Lincoln Laboratory
Massachusetts Institute of Technology. Department of Materials Science and Engineering
Massachusetts Institute of Technology. Department of Chemical Engineering
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution
https://creativecommons.org/licenses/by/4.0/
Persistent DSpace Link
https://hdl.handle.net/1721.1/158195
DOI of Published Version
https://doi.org/10.1038/s42256-023-00740-3
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.