A perturbative analysis of stochastic descent
Name
1227278188-MIT.pdf
Size
3.04 MB
Format
Adobe PDF
Checksum (MD5)
52cd64ae71e046be220e54139c1485bf
Author(s)
Tenka, Samuel C.
Advisor(s)
Joshua B. Tenenbaum.
Date Issued
2020
Publisher
Massachusetts Institute of Technology
Abstract
We analyze stochastic gradient descent (SGD) at small learning rates. Unlike prior analyses based on stochastic differential equations, our theory models discrete time and hence non-Gaussian noise. We illustrate our theory by discussing four of its corollaries: we (A) generalize the Akaike information criterion (AIC) to a smooth estimator of overfitting, hence enabling gradient-based model selection; (B) show how non-stochastic GD with a modified loss function may emulate SGD; (C) prove that gradient noise systematically pushes SGD toward flatter minima; and (D) characterize when and why flat minima overfit less than other minima.
Description
Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, September, 2020
Cataloged from student-submitted PDF version of thesis.
Includes bibliographical references.
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided.
Persistent DSpace Link
