Comparing rewinding and fine-tuning in neural network pruning
Name
1192486982-MIT.pdf
Size
1.92 MB
Format
Adobe PDF
Checksum (MD5)
2320a194d11f3eaae6bdd1751ea8d130
Author(s)
Renda, Alex(Alexander Dominic)
Advisor(s)
Michael Carbin.
Date Issued
2020
Publisher
Massachusetts Institute of Technology
Abstract
Many neural network pruning algorithms proceed in three steps: train the network to completion, remove unwanted structure to compress the network, and retrain the remaining structure to recover lost accuracy. The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate. In this thesis, I compare fine-tuning to alternative retraining techniques. Weight rewinding (as proposed by Frankle et al. (2019)), rewinds unpruned weights to their values from earlier in training and retrains them from there using the original training schedule. Learning rate rewinding (proposed in this thesis) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding. Both rewinding techniques outperform fine-tuning, forming the basis of a network-agnostic pruning algorithm that matches the accuracy and compression ratios of several more network-specific state-of-the-art techniques.
Description
Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, May, 2020
Cataloged from the official PDF of thesis.
Includes bibliographical references (pages 40-44).
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided.
Persistent DSpace Link
