On Neural Network Pruning’s Effect on Generalization
Name
Jin-tianjin-SM-EECS-2022-thesis.pdf.pdf
Description
Thesis PDF
Size
14.58 MB
Format
Adobe PDF
Checksum (MD5)
785024465c6acef6ea1f4416b1256d4a
Author(s)
Jin, Tian
Advisor(s)
Carbin, Michael
Date Issued
September 2022
Publisher
Massachusetts Institute of Technology
Abstract
Practitioners frequently observe that pruning improves model generalization. A longstanding hypothesis attributes such improvement to model size reduction. However, recent studies on over-parameterization characterize a new model size regime, in which larger models achieve better generalization. A contradiction arises when pruning is applied to over-parameterized models – while theory predicts that reducing size harms generalization, pruning nonetheless improves it. Motivated by such a contradiction, I re-examine pruning’s effect on generalization empirically.
I demonstrate that pruning’s generalization-improving effect cannot be fully accounted for by weight removal. Instead, I find that pruning can lead to better training, improving model training loss. I find that pruning can also lead to stronger regularization, mitigating the harmful effect of noisy examples. Pruning extends model training time and reduces model size, which improves training and strengthens regularization respectively. I empirically demonstrate that both factors are essential to explaining pruning’s benefits to generalization fully.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
