Fisher-rao metric, geometry, and complexity of neural networks
Name
liang19a.pdf
Description
Published version
Size
909.73 KB
Format
Adobe PDF
Checksum (MD5)
504f65d125e54b88d9715321a20d33f1
Author(s)
Liang, T
Poggio, Tomaso A
Rakhlin, A
Stokes, J
Date Issued
January 1, 2020
Journal
AISTATS 2019 - 22nd International Conference on Artificial Intelligence and Statistics
Citation
Liang, T, Poggio, T, Rakhlin, A and Stokes, J. 2020. "Fisher-rao metric, geometry, and complexity of neural networks." AISTATS 2019 - 22nd International Conference on Artificial Intelligence and Statistics, 89.
Version
Final published version
Abstract
© 2019 by the author(s). We study the relationship between geometry and capacity measures for deep neural networks from an invariance viewpoint. We introduce a new notion of capacity - the Fisher-Rao norm - that possesses desirable invariance properties and is motivated by Information Geometry. We discover an analytical characterization of the new capacity measure, through which we establish norm-comparison inequalities and further show that the new measure serves as an umbrella for several existing norm-based complexity measures. We discuss upper bounds on the generalization error induced by the proposed measure. Extensive numerical experiments on CIFAR-10 support our theoretical findings. Our theoretical analysis rests on a key structural lemma about partial derivatives of multi-layer rectifier networks.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
McGovern Institute for Brain Research at MIT
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Statistics and Data Science Center (Massachusetts Institute of Technology)
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
http://proceedings.mlr.press/v89/liang19a/liang19a.pdf
