Compositional Sparsity of Learnable Functions
Name
CBMM-Memo-145.pdf
Size
1.25 MB
Format
Adobe PDF
Checksum (MD5)
8c8eba1f527772a22dc687dfa336ad3a
Author(s)
Poggio, Tomaso
Fraser, Maia
Date Issued
February 8, 2024
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;145
Abstract
Neural networks have demonstrated impressive success in various domains, raising the question of what fundamental principles underlie the effectiveness of the best AI systems and quite possibly of human intelligence. This perspective argues that compositional sparsity, or the property that a compositional function have "few" constituent functions, each depending on only a small subset of inputs, is a key principle underlying successful learning architectures. Surprisingly, all functions that are efficiently Turing computable have a compositional sparse representation. Furthermore, deep networks that are also sparse can exploit this general property to avoid the “curse of dimensionality". This framework suggests interesting implications about the role that machine learning may play in mathematics.
Persistent DSpace Link
