An Information Theoretic Interpretation to Deep Neural Networks
Name
entropy-24-00135-v2.pdf
Size
497.87 KB
Format
Adobe PDF
Checksum (MD5)
7a04374d15852df383973e0d76de044b
Author(s)
Xu, Xiangxiang
Huang, Shao-Lun
Zheng, Lizhong
Wornell, Gregory W.
Date Issued
January 17, 2022
Publisher
Multidisciplinary Digital Publishing Institute
Citation
Entropy 24 (1): 135 (2022)
Version
Final published version
Abstract
With the unprecedented performance achieved by deep learning, it is commonly believed that deep neural networks (DNNs) attempt to extract informative features for learning tasks. To formalize this intuition, we apply the local information geometric analysis and establish an information-theoretic framework for feature selection, which demonstrates the information-theoretic optimality of DNN features. Moreover, we conduct a quantitative analysis to characterize the impact of network structure on the feature extraction process of DNNs. Our investigation naturally leads to a performance metric for evaluating the effectiveness of extracted features, called the H-score, which illustrates the connection between the practical training process of DNNs and the information-theoretic framework. Finally, we validate our theoretical results by experimental designs on synthesized data and the ImageNet dataset.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.3390/e24010135
