An efficient algorithm for information decomposition and extraction
Name
Zheng_An efficient.pdf
Size
318.73 KB
Format
Adobe PDF
Checksum (MD5)
d5d8f6cdc990bbdb05d72410d62bf511
Author(s)
Makur, Anuran
Kozynski Waserman, Fabian Ariel
Huang, Shao-Lun
Zheng, Lizhong
Date Issued
September 2015
Journal
2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton)
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Makur, Anuran et al.. “An Efficient Algorithm for Information Decomposition and Extraction.” 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton) 29 September - 2 October, 2015, Monticello, Illinois, 2015.
Version
Author's final manuscript
Abstract
The Hirschfeld-Gebelein-Rényi maximal correlation is a well-known measure of statistical dependence between two (possibly categorical) random variables. In inference problems, the maximal correlation functions can be viewed as so called features of observed data that carry the largest amount of information about some latent variables. These features are in general non-linear functions, and are particularly useful in processing high-dimensional observed data. The alternating conditional expectations (ACE) algorithm is an efficient way to compute these maximal correlation functions. In this paper, we use an information theoretic approach to interpret the ACE algorithm as computing the singular value decomposition of a linear map between spaces of probability distributions. With this approach, we demonstrate the information theoretic optimality of the ACE algorithm, analyze its convergence rate and sample complexity, and finally, generalize it to compute multiple pairs of correlation functions from samples.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/ALLERTON.2015.7447113
