An Information Theoretic Interpretation to Deep Neural Networks
Name
1905.06600.pdf
Description
Submitted version
Size
681.74 KB
Format
Adobe PDF
Checksum (MD5)
865aacb13d9aabfa0ea1b0fce413b09d
Author(s)
Huang, Shao-Lun
Xu, Xiangxiang
Zheng, Lizhong
Wornell, Gregory W
Date Issued
July 2019
Journal
IEEE International Symposium on Information Theory - Proceedings
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Huang, Shao-Lun, Xu, Xiangxiang, Zheng, Lizhong and Wornell, Gregory W. 2019. "An Information Theoretic Interpretation to Deep Neural Networks." IEEE International Symposium on Information Theory - Proceedings, 2019-July.
Version
Original manuscript
Abstract
© 2019 IEEE. It is commonly believed that the hidden layers of deep neural networks (DNNs) attempt to extract informative features for learning tasks. In this paper, we formalize this intuition by showing that the features extracted by DNN coincide with the result of an optimization problem, which we call the "universal feature selection" problem, in a local analysis regime. We interpret the weights training in DNN as the projection of feature functions between feature spaces, specified by the network structure. Our formulation has direct operational meaning in terms of the performance for inference tasks, and gives interpretations to the internal computation results of DNNs. Results of numerical experiments are provided to support the analysis.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/ISIT.2019.8849720
