2026-09-19T00:00:37Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1218302026-06-17T14:44:25Zcom_1721.1_7582com_1721.1_7581col_1721.1_131022
Suvrit Sra.
Zhang, Hongyi,Ph. D.Massachusetts Institute of Technology.
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences.
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
2019-07-18T20:34:28Z
2019-07-18T20:34:28Z
2019
2019
https://hdl.handle.net/1721.1/121830
1108619914
Thesis: Ph. D., Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences, 2019
Cataloged from PDF version of thesis.
Includes bibliographical references (pages 165-186).
Non-convex optimization and learning play an important role in data science and machine learning, yet so far they still elude our understanding in many aspects. In this thesis, I study two important aspects of non-convex optimization and learning: Riemannian optimization and deep neural networks. In the first part, I develop iteration complexity analysis for Riemannian optimization, i.e., optimization problems defined on Riemannian manifolds. Through bounding the distortion introduced by the metric curvature, iteration complexity of Riemannian (stochastic) gradient descent methods is derived. I also show that some fast first-order methods in Euclidean space, such as Nesterov's accelerated gradient descent (AGD) and stochastic variance reduced gradient (SVRG), have Riemannian counterparts that are also fast under certain conditions. In the second part, I challenge two common practices in deep learning, namely empirical risk minimization (ERM) and normalization. Specifically, I show (1) training on convex combinations of samples improves model robustness and generalization, and (2) a good initialization is sufficient for training deep residual networks without normalization. The method in (1), called mixup, is motivated by a data-dependent Lipschitzness regularization of the network. The method in (2), called Zerolnit, makes the network update scale invariant to its depth at initialization.
by Hongyi Zhang.
Ph.D.
Ph.D. Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences
186 pages
eng
Massachusetts Institute of Technology
MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.
http://dspace.mit.edu/handle/1721.1/7582
Brain and Cognitive Sciences.
Topics in non-convex optimization and learning
Thesis
application/pdf
2019-07-18T20:34:25Z
Publication
Doctoral
Brain
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="8eb27a69-4fd2-466f-8808-22c4978ed4eb">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Language>eng</Language>
<Title>Topics in non-convex optimization and learning</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2019</PublicationDate>
<Authors>
<Author>
<DisplayName>Zhang, Hongyi,Ph. D.Massachusetts Institute of Technology.</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://dspace.mit.edu/handle/1721.1/7582</License>
<Keyword>Brain and Cognitive Sciences.</Keyword>
<Abstract>Non-convex optimization and learning play an important role in data science and machine learning, yet so far they still elude our understanding in many aspects. In this thesis, I study two important aspects of non-convex optimization and learning: Riemannian optimization and deep neural networks. In the first part, I develop iteration complexity analysis for Riemannian optimization, i.e., optimization problems defined on Riemannian manifolds. Through bounding the distortion introduced by the metric curvature, iteration complexity of Riemannian (stochastic) gradient descent methods is derived. I also show that some fast first-order methods in Euclidean space, such as Nesterov's accelerated gradient descent (AGD) and stochastic variance reduced gradient (SVRG), have Riemannian counterparts that are also fast under certain conditions. In the second part, I challenge two common practices in deep learning, namely empirical risk minimization (ERM) and normalization. Specifically, I show (1) training on convex combinations of samples improves model robustness and generalization, and (2) a good initialization is sufficient for training deep residual networks without normalization. The method in (1), called mixup, is motivated by a data-dependent Lipschitzness regularization of the network. The method in (2), called Zerolnit, makes the network update scale invariant to its depth at initialization.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>