Alternating conditional expectation (ACE) applied to classification and recommendation problems

Abstract

In this thesis, a geometric framework for describing relevant information in a collection of data is applied for the general problems of selecting informative features (dimension reduction) from high dimensional data. The framework can be used in an unsupervised manner, extracting universal features that can be used later for general classification of data. This framework is derived by applying local approximations on the space of probability distributions and a small perturbation approach. With this approach, different information theoretic results can be interpreted as linear algebra optimizations based on the norms of vectors in a linear space, which are in general, easier to carry out. Fundamentally, using known procedures such as Singular Value Decomposition (SVD) and Principal Component Analysis (PCA), dimension reduction for maximizing power can be achieved in a straight forward manner. Using the geometric framework, we relate calculation of SVD of a particular matrix related to a probabilistic channel to the application of Alternating Conditional Expectation (ACE) in the problem of optimal regression. The key takeaway of this method is that such problems can be studied in the space of distributions of the data and not the space of outcomes. This geometric framework allows to give an operational meaning to information metrics in the context of data analysis and feature selection. Additionally, it provides a method to obtain universal classification functions without knowledge of the important feature of the problem. This framework is the applied to the problem of data classification and analysis with satisfactory results.