2026-09-18T22:20:55Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1513142023-08-01T03:41:15Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023

Azizan, Navid Huang, Tiffany Y. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2023-07-31T19:30:40Z 2023-07-31T19:30:40Z 2023-06 2023-06-06T16:34:39.910Z https://hdl.handle.net/1721.1/151314 In many learning problems, it is desirable to incorporate explicit regularization in the objective to avoid overfitting the data. Typically, the regularized objective is solved via weight decay. However, optimizing with weight decay can be challenging because we cannot tell if the solution has reached a global minimum. Further, weight decay can have large run-to-run variations and is sensitive to the choice of regularization hyperparameter. To this end, we propose a new approach to optimize objectives with explicit regularization, called Regularizer Mirror Descent (RMD). In the overparameterized regime, where the number of model parameters exceeds the size of data, RMD provably converges to a point “close” to a minimizer of the regularized objective. Additionally, RMD is computationally efficient and imposes virtually no overhead to standard gradient descent. We observe that RMD is remarkably robust and consistent compared to gradient descent with weight decay despite solving for the same objective. We also illustrate the practical utility of RMD by applying it to learning problems with corrupted labels, where it can match or outperform the state-of-the-art methods without requiring additional hyperparameter tuning or ad-hoc heuristics tailored for this task. M.Eng. Massachusetts Institute of Technology In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ Explicit Regularization for Overparameterized Models Thesis application/pdf Master Master of Engineering in Electrical Engineering and Computer Science Publication unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="8d71d834-c026-4456-ae57-084a85eda02f"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Title>Explicit Regularization for Overparameterized Models</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2023-06</PublicationDate> <Authors> <Author> <DisplayName>Huang, Tiffany Y.</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>https://rightsstatements.org/page/InC-EDU/1.0/</License> <Abstract>In many learning problems, it is desirable to incorporate explicit regularization in the objective to avoid overfitting the data. Typically, the regularized objective is solved via weight decay. However, optimizing with weight decay can be challenging because we cannot tell if the solution has reached a global minimum. Further, weight decay can have large run-to-run variations and is sensitive to the choice of regularization hyperparameter. To this end, we propose a new approach to optimize objectives with explicit regularization, called Regularizer Mirror Descent (RMD). In the overparameterized regime, where the number of model parameters exceeds the size of data, RMD provably converges to a point “close” to a minimizer of the regularized objective. Additionally, RMD is computationally efficient and imposes virtually no overhead to standard gradient descent. We observe that RMD is remarkably robust and consistent compared to gradient descent with weight decay despite solving for the same objective. We also illustrate the practical utility of RMD by applying it to learning problems with corrupted labels, where it can match or outperform the state-of-the-art methods without requiring additional hyperparameter tuning or ad-hoc heuristics tailored for this task.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>