Block sparsity and weight initialization in neural network pruning
Name
1251801573-MIT.pdf
Size
14.65 MB
Format
Adobe PDF
Checksum (MD5)
8640c24930f6d21ef4d333daff585531
Author(s)
Siswanto, Arlene Elizabeth.
Advisor(s)
Michael Carbin.
Date Issued
2021
Publisher
Massachusetts Institute of Technology
Abstract
Block sparsity imposes structural constraints on the weight patterns of sparse neural networks. The structure of sparsity has been shown to affect efficiency of sparse computation in the libraries, kernels, and hardware commonly used in machine learning. Much work in the pruning literature has focused on the unstructured pruning of individual weights, which has been shown to reduce the memory footprint of a network, but cannot achieve the computational speedups that have become increasingly coveted as neural networks become deeper and more complex. On the opposite end of granularity, neuron pruning and channel pruning are unable to reach the same level of sparsity as unstructured pruning without compromising accuracy. Block-sparse pruning is a middle ground between these two extremes, with the potential for pruning to greater sparsities while still being amenable for acceleration. Our fine-tuning experiments demonstrate that block-sparse pruning offers a tradeoff between granularity and accuracy; increasing block size results in a gradual decrease in accuracy. Our weight rewinding experiments show that increasing block size decreases the maximum sparsity obtainable when pruning a network early in training. Finally, we make the surprising observation that randomly reinitializing the pruned network structure results in the same accuracy regardless of block size.
Description
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, February, 2021
Cataloged from the official PDF of thesis.
Includes bibliographical references (pages 69-72).
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided.
Persistent DSpace Link
