2026-09-19T15:23:54Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1230282026-06-06T00:49:08Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023

Aleksander Ma̧dry. Wei, Wendy,M. Eng.Massachusetts Institute of Technology. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2019-11-22T00:03:08Z 2019-11-22T00:03:08Z 2019 2019 https://hdl.handle.net/1721.1/123028 1127649566 This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2019 Cataloged from student-submitted PDF version of thesis. Includes bibliographical references (pages 29-30). Deep neural networks are known to be highly non-convex. Many of the methods used in deep learning which are informed by convex optimization work surprisingly well. The training dynamics of optimization methods such as momentum suggest that training occurs in distinct regimes, attributed to learning rate. In the low learning rate regime, many convex intuitions hold, and the recommended methods are able to reach a good solution. In the high learning rate regime, the training behavior is not convex-like, but training longer in this period achieves better generalization. This thesis focuses on rethinking deep network training from the perspective of these phases in training. Empirical results suggest that each training regime, although distinct, work together to produce high performance on deep learning tasks. Moreover, we re-examine popular learning rate schedules and find that the paradigm of high and low learning rate regimes helps to explain their advantages. by Wendy Wei. M.Eng. M.Eng. Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science 30 pages eng Massachusetts Institute of Technology MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582 Electrical Engineering and Computer Science. Rethinking methods to train deep neural networks : contributions of distinct regimes during training Thesis application/pdf 2019-11-22T00:03:07Z Publication Master EECS unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="2fb5ee60-cf72-4028-b871-3e626ad98557"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>eng</Language> <Title>Rethinking methods to train deep neural networks : contributions of distinct regimes during training</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2019</PublicationDate> <Authors> <Author> <DisplayName>Wei, Wendy,M. Eng.Massachusetts Institute of Technology.</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>http://dspace.mit.edu/handle/1721.1/7582</License> <Keyword>Electrical Engineering and Computer Science.</Keyword> <Abstract>Deep neural networks are known to be highly non-convex. Many of the methods used in deep learning which are informed by convex optimization work surprisingly well. The training dynamics of optimization methods such as momentum suggest that training occurs in distinct regimes, attributed to learning rate. In the low learning rate regime, many convex intuitions hold, and the recommended methods are able to reach a good solution. In the high learning rate regime, the training behavior is not convex-like, but training longer in this period achieves better generalization. This thesis focuses on rethinking deep network training from the perspective of these phases in training. Empirical results suggest that each training regime, although distinct, work together to produce high performance on deep learning tasks. Moreover, we re-examine popular learning rate schedules and find that the paradigm of high and low learning rate regimes helps to explain their advantages.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>