Deep vs. shallow networks: An approximation theory perspective
Name
1608.03287.pdf
Description
Accepted version
Size
960.27 KB
Format
Adobe PDF
Checksum (MD5)
b3df910478058b77adc2db162c167268
Author(s)
Mhaskar, HN
Poggio, Tomaso A
Date Issued
2016
Journal
Analysis and Applications
Publisher
World Scientific Pub Co Pte Lt
Citation
Mhaskar, H. N., and T. Poggio. "Deep Vs. Shallow Networks: An Approximation Theory Perspective." Analysis and Applications 14 6 (2016): 829-48.
Version
Author's final manuscript
Abstract
© 2016 World Scientific Publishing Company. The paper briefly reviews several recent results on hierarchical architectures for learning from examples, that may formally explain the conditions under which Deep Convolutional Neural Networks perform much better in function approximation problems than shallow, one-hidden layer architectures. The paper announces new results for a non-smooth activation function - the ReLU function - used in present-day neural networks, as well as for the Gaussian networks. We propose a new definition of relative dimension to encapsulate different notions of sparsity of a function class that can possibly be exploited by deep networks but not by shallow ones to drastically reduce the complexity required for approximation and learning.
MIT Department
Center for Brains, Minds, and Machines
McGovern Institute for Brain Research at MIT
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
10.1142/S0219530516400042
https://doi.org/10.1142/S0219530516400042
