2026-09-19T09:29:54Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1627222025-09-19T04:49:38Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Mądry, Aleksander
Chen, Benjamin
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2025-09-18T14:29:04Z
2025-09-18T14:29:04Z
2025-05
2025-06-23T14:01:20.449Z
https://hdl.handle.net/1721.1/162722
A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
Metagradient Descent: Differentiating Large-Scale Training
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="cd6f1d71-0c19-4810-9bf2-3ffe2da07a2f">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Metagradient Descent: Differentiating Large-Scale Training</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2025-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Chen, Benjamin</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>