Analyzing Multimodal Interactions through Improved Partial Information Decomposition Estimation
Name
balachandran-adithyab-meng-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
13.48 MB
Format
Adobe PDF
Checksum (MD5)
86abefb84ddbf7357d0fb3efd0b981da
Author(s)
Balachandran, Adithya S.
Advisor(s)
Liang, Paul Pu
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
Multimodal AI aims to build comprehensive models by integrating information from diverse sensory inputs such as text, audio, and vision. However, significant challenges remain in understanding how these different modalities interact and contribute to downstream tasks. In particular, we seek to characterize how modalities complement each other, overlap in the information they convey, or contribute jointly to patterns that are not clear from any single modality alone. To address this, we propose novel methods for quantifying these multimodal interactions using information-theoretic techniques. Specifically, we will introduce a novel estimator for Partial Information Decomposition (PID) using normalizing flows, with the ability to scale well to high-dimensional data. We also develop a new framework for estimating pointwise PID, which provides insights into how individual data points contribute to information sharing and interactions across modalities, and show how to apply this framework for anomaly detection. We demonstrate the effectiveness of our methods on a variety of high-dimensional datasets, including both synthetic and real-world multimodal data such as videos.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
