Statistical modeling and analysis of audio-visual association in speech

Abstract

Currently, most dialog systems are restricted to single user environments. This thesis aims to promote an un-tethered multi-person dialog system by exploring approaches to help solve the speech correspondence problem (i.e. who, if anyone, is currently speaking). We adopt a statistical framework in which this problem is put in the form of a hypothesis test and focus on the subtask of discriminating between associated and non-associated audio-visual observations. Various methods for modeling our audio-visual observations and ways of carrying out this test are studied and their relative performance is compared. We discuss issues that arise from the inherently high dimensional nature of audio-visual data and address these issues by exploring different techniques for finding low-dimensional informative subspaces in which we can perform our hypothesis tests. We study our ability to learn a person-specific as well as a generic model for measuring audio-visual association and evaluate performance oil multiple subjects taken from MIT's AVTIMIT database.