I-theory on depth vs width: hierarchical function composition
Name
CBMM-Memo-041.pdf
Size
1.18 MB
Format
Adobe PDF
Checksum (MD5)
f74599076886ec07eeee7ee06a0744d3
Author(s)
Poggio, Tomaso
Anselmi, Fabio
Rosasco, Lorenzo
Date Issued
December 29, 2015
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo Series;041
Abstract
Deep learning networks with convolution, pooling and subsampling are a special case of hierar- chical architectures, which can be represented by trees (such as binary trees). Hierarchical as well as shallow networks can approximate functions of several variables, in particular those that are com- positions of low dimensional functions. We show that the power of a deep network architecture with respect to a shallow network is rather independent of the specific nonlinear operations in the network and depends instead on the the behavior of the VC-dimension. A shallow network can approximate compositional functions with the same error of a deep network but at the cost of a VC-dimension that is exponential instead than quadratic in the dimensionality of the function. To complete the argument we argue that there exist visual computations that are intrinsically compositional. In particular, we prove that recognition invariant to translation cannot be computed by shallow networks in the presence of clutter. Finally, a general framework that includes the compositional case is sketched. The key con- dition that allows tall, thin networks to be nicer that short, fat networks is that the target input-output function must be sparse in a certain technical sense.
Subjects
Deep Convolutional Learning Networks (DCLNs)
Hierarchy
i-theory
Terms of Use
Attribution-NonCommercial 3.0 United States
Persistent DSpace Link
