ResNet with one-neuron hidden layers is a Universal Approximator
Name
NeurIPS-2018-resnet-with-one-neuron-hidden-layers-is-a-universal-approximator-Paper.pdf
Description
Published version
Size
734.66 KB
Format
Adobe PDF
Checksum (MD5)
c899097ca54372ce1ce9a29a05b908b4
Author(s)
Lin, Hongzhou
Jegelka, Stefanie Sabrina
Date Issued
December 2018
Journal
Advances in Neural Information Processing Systems
Publisher
Morgan Kaufmann Publishers
Citation
Lin, Hongzhou and Stefanie Jegelka. “ResNet with one-neuron hidden layers is a Universal Approximator.” Advances in Neural Information Processing Systems, December-2018 (December 2018) © 2018 The Author(s)
Version
Final published version
Abstract
We demonstrate that a very deep ResNet with stacked modules that have one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in d dimensions, i.e. ℓ1(Rd). Due to the identity mapping inherent to ResNets, our network has alternating layers of dimension one and d. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension d [21, 11]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://papers.nips.cc/paper/2018/hash/03bfc1d4783966c69cc6aef8247e0103-Abstract.html
