Small ReLU networks are powerful memorizers: A tight analysis of memorization capacity
Name
NeurIPS-2019-small-relu-networks-are-powerful-memorizers-a-tight-analysis-of-memorization-capacity-Paper.pdf
Description
Published version
Size
311.17 KB
Format
Adobe PDF
Checksum (MD5)
37175e9ecf0410af49f518124bafa7b7
Author(s)
Yun, Chulhee
Sra, Suvrit
Jadbabaie, Ali
Date Issued
2019
Journal
Advances in Neural Information Processing Systems
Citation
Yun, Chulhee, Sra, Suvrit and Jadbabaie, Ali. 2019. "Small ReLU networks are powerful memorizers: A tight analysis of memorization capacity." Advances in Neural Information Processing Systems, 32.
Version
Final published version
Abstract
© 2019 Neural information processing systems foundation. All rights reserved. We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require N hidden nodes to memorize/interpolate arbitrary N data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with ?(vN) hidden nodes can perfectly memorize most datasets with N points. We also prove that width T(vN) is necessary and sufficient for memorizing N data points, proving tight bounds on memorization capacity. The sufficiency result can be extended to deeper networks; we show that an L-layer network with W parameters in the hidden layers can memorize N data points if W = ?(N). Combined with a recent upper bound O(WLlog W) on VC dimension, our construction is nearly tight for any fixed L. Subsequently, we analyze memorization capacity of residual networks under a general position assumption; we prove results that substantially reduce the known requirement of N hidden nodes. Finally, we study the dynamics of stochastic gradient descent (SGD), and show that when initialized near a memorizing global minimum of the empirical risk, SGD quickly finds a nearby point with much smaller empirical risk.
MIT Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://papers.nips.cc/paper/2019/hash/dbea3d0e2a17c170c412c74273778159-Abstract.html
