From Associative Memories to Deep Networks
Name
CBMM-Memo-114.pdf
Size
1.01 MB
Format
Adobe PDF
Checksum (MD5)
2392a54a3e34d78481a6f8110c8a2adf
Author(s)
Poggio, Tomaso
Date Issued
January 12, 2021
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;114
Abstract
About fifty years ago, holography was proposed as a model of associative memory. Associative memories with similar properties were soon after implemented as simple networks of threshold neurons by Willshaw and Longuet-Higgins. In these pages I will show that today’s deep nets are an incremental improvement of the original associative networks. Thinking about deep learning in terms of associative networks provides a more realistic and sober perspective on the promises of deep learning and on its role in eventually understanding human intelligence. As a bonus, this discussion also uncovers connections with several interesting topics in applied math: random features, random projections, neural ensembles, randomized kernels, memory and generalization, vector quantization and hierarchical vector quantization, random vectors and orthogonal basis, NTK and radial kernels.
Persistent DSpace Link
