Overparameterized neural networks implement associative memory
Name
27162.full.pdf
Description
Published version
Size
15.28 MB
Format
Adobe PDF
Checksum (MD5)
003b11cfef59497b4706f582fe0ab82f
Author(s)
Radhakrishnan, Adityanarayanan
Belkin, Mikhail
Uhler, Caroline
Date Issued
2020
Journal
Proceedings of the National Academy of Sciences of the United States of America
Publisher
Proceedings of the National Academy of Sciences
Version
Final published version
Abstract
© 2020 National Academy of Sciences. All rights reserved. Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. We provide empirical evidence that 1) overparameterized autoencoders store training samples as attractors and thus iterating the learned map leads to sample recovery, and that 2) the same mechanism allows for encoding sequences of examples and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.
MIT Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1073/PNAS.2005013117
