Rethinking methods to train deep neural networks : contributions of distinct regimes during training
Name
1127649566-MIT.pdf
Size
1004.28 KB
Format
Adobe PDF
Checksum (MD5)
4eb8e70907414348798edc22b7b17d53
Author(s)
Wei, Wendy,M. Eng.Massachusetts Institute of Technology.
Advisor(s)
Aleksander Ma̧dry.
Date Issued
2019
Publisher
Massachusetts Institute of Technology
Abstract
Deep neural networks are known to be highly non-convex. Many of the methods used in deep learning which are informed by convex optimization work surprisingly well. The training dynamics of optimization methods such as momentum suggest that training occurs in distinct regimes, attributed to learning rate. In the low learning rate regime, many convex intuitions hold, and the recommended methods are able to reach a good solution. In the high learning rate regime, the training behavior is not convex-like, but training longer in this period achieves better generalization. This thesis focuses on rethinking deep network training from the perspective of these phases in training. Empirical results suggest that each training regime, although distinct, work together to produce high performance on deep learning tasks. Moreover, we re-examine popular learning rate schedules and find that the paradigm of high and low learning rate regimes helps to explain their advantages.
Description
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2019
Cataloged from student-submitted PDF version of thesis.
Includes bibliographical references (pages 29-30).
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.
Persistent DSpace Link
