Metagradient Descent: Differentiating Large-Scale Training
Name
chen-benchen-meng-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
1.19 MB
Format
Adobe PDF
Checksum (MD5)
b9289d29fc09375d276a97db4f8630e7
Author(s)
Chen, Benjamin
Advisor(s)
Mądry, Aleksander
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
A major challenge in training large-scale machine learning models is configuring the training process to maximize model performance, i.e., finding the best training setup from a vast design space. In this work, we unlock a gradient-based approach to this problem. We first introduce an algorithm for efficiently calculating metagradients -- gradients through model training -- at scale. We then introduce a "smooth model training" framework that enables effective optimization using metagradients. With metagradient descent (MGD), we greatly improve on existing dataset selection methods, outperform accuracy-degrading data poisoning attacks by an order of magnitude, and automatically find competitive learning rate schedules.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
