Constrained Information Exchange Message Passing Algorithm in Probabilistic Graphical Models

Abstract

Probabilistic graphical models combine probability and graph theory into a powerful multivariate statistical modeling approach. While there is an extraordinary range of types of graphical models in the broader literature, we focus on Hidden Markov Model (HMM) and Linear Dynamical System (LDS). These two models are ubiquitous in applications including Kalman filtering, and the decoding of LDPC codes which is what all cellular wireless systems run on these days. Message passing algorithms exploit the independence and factorization structure within these graphical models to develop analytically tractable, computationally efficient, and exact inference algorithms. Posterior inference using message-passing algorithm depends heavily on the information exchange between the nodes of the graph and having unconstrained sized messages, results in the exact inference of the posterior distribution. Despite its usefulness and ubiquitous applications, the unconstrained information exchange message-passing algorithm has numerous shortcomings, including prohibitive computational complexity, and unaffordable communication complexity. These shortcomings can limit the effectiveness and practicality of the algorithm in the era of big data.

This thesis presents a comprehensive analysis of a novel constrained information exchange message-passing algorithm. A cornerstone of this algorithm is the modified sum product algorithm, an innovative approach that identifies and prioritizes critical information fragments for particular inference tasks.

The thesis further delves into the algorithm’s utility in various posterior inference scenarios, encompassing index-specific and index-free posterior inferences. While the former scrutinizes either singular or multiple posterior inferences at designated indexes, the latter ensures that all nodes use identical compression matrices, catering to both single and multi-step posterior inferences.

The algorithm elucidates the balance between algorithmic performance and resource utilization in data-driven applications. By reducing computational and communication complexities, it proposes an efficient solution for high-dimensional data handling, particularly when resources are constrained. In essence, the proposed algorithm enhances the scalability, practicality, and efficiency of probabilistic graphical models, propelling advancements in data-driven research and decision-making across a multitude of domains.