2026-09-19T04:47:21Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1474962023-01-20T03:15:28Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Carbin, Michael
Jin, Tian
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2023-01-19T19:54:15Z
2023-01-19T19:54:15Z
2022-09
2022-10-19T18:57:25.582Z
https://hdl.handle.net/1721.1/147496
Practitioners frequently observe that pruning improves model generalization. A longstanding hypothesis attributes such improvement to model size reduction. However, recent studies on over-parameterization characterize a new model size regime, in which larger models achieve better generalization. A contradiction arises when pruning is applied to over-parameterized models – while theory predicts that reducing size harms generalization, pruning nonetheless improves it. Motivated by such a contradiction, I re-examine pruning’s effect on generalization empirically.
I demonstrate that pruning’s generalization-improving effect cannot be fully accounted for by weight removal. Instead, I find that pruning can lead to better training, improving model training loss. I find that pruning can also lead to stronger regularization, mitigating the harmful effect of noisy examples. Pruning extends model training time and reduces model size, which improves training and strengthens regularization respectively. I empirically demonstrate that both factors are essential to explaining pruning’s benefits to generalization fully.
S.M.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright MIT
http://rightsstatements.org/page/InC-EDU/1.0/
On Neural Network Pruning’s Effect on Generalization
Thesis
application/pdf
Master
Master of Science in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="be9e639a-d6df-40b2-bb9b-2738cd02aafd">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>On Neural Network Pruning’s Effect on Generalization</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2022-09</PublicationDate>
<Authors>
<Author>
<DisplayName>Jin, Tian</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Practitioners frequently observe that pruning improves model generalization. A longstanding hypothesis attributes such improvement to model size reduction. However, recent studies on over-parameterization characterize a new model size regime, in which larger models achieve better generalization. A contradiction arises when pruning is applied to over-parameterized models – while theory predicts that reducing size harms generalization, pruning nonetheless improves it. Motivated by such a contradiction, I re-examine pruning’s effect on generalization empirically.
I demonstrate that pruning’s generalization-improving effect cannot be fully accounted for by weight removal. Instead, I find that pruning can lead to better training, improving model training loss. I find that pruning can also lead to stronger regularization, mitigating the harmful effect of noisy examples. Pruning extends model training time and reduces model size, which improves training and strengthens regularization respectively. I empirically demonstrate that both factors are essential to explaining pruning’s benefits to generalization fully.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>