Data-dependent coresets for compressing neural networks with applications to generalization bounds
Name
data_dependent_coresets_for_compressing_neural_networks_with_applications_to_generalization_bounds.pdf
Description
Accepted version
Size
1.78 MB
Format
Unknown
Checksum (MD5)
f16323e54f00031c416444fd8e1de9ae
Author(s)
Baykal, Cenk
Liebenwein, Lucas
Gilitschenski, Igor
Feldman, Dan
Rus, Daniela L
Date Issued
May 2019
Journal
7th International Conference on Learning Representations, ICLR 2019
Citation
2019. "Data-dependent coresets for compressing neural networks with applications to generalization bounds." 7th International Conference on Learning Representations, ICLR 2019.
Version
Author's final manuscript
Abstract
We present an efficient coresets-based neural network compression algorithm that sparsifies the parameters of a trained fully-connected neural network in a manner that provably approximates the network's output. Our approach is based on an importance sampling scheme that judiciously defines a sampling distribution over the neural network parameters, and as a result, retains parameters of high importance while discarding redundant ones. We leverage a novel, empirical notion of sensitivity and extend traditional coreset constructions to the application of compressing parameters. Our theoretical analysis establishes guarantees on the size and accuracy of the resulting compressed network and gives rise to generalization bounds that may provide new insights into the generalization properties of neural networks. We demonstrate the practical effectiveness of our algorithm on a variety of neural network configurations and real-world data sets.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
