Improving automatic speech recognition through head pose driven visual grounding
Name
CHI2014_Vosoughi.pdf
Description
Main Article
Size
813.48 KB
Format
Adobe PDF
Checksum (MD5)
28ed3b18469c8ad50c6a9396807fe4e9
Author(s)
Vosoughi, Soroush
Date Issued
2014
Journal
Proceedings of the 32nd annual ACM conference on Human factors in computing systems - CHI '14
Publisher
Association for Computing Machinery
Citation
Vosoughi, Soroush. “Improving Automatic Speech Recognition through Head Pose Driven Visual Grounding.” Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems - CHI ’14 (2014), April 26–May 01, 2014, Toronto, ON, Canada.
Version
Final published version
Abstract
In this paper, we present a multimodal speech recognition system for real world scene description tasks. Given a visual scene, the system dynamically biases its language model based on the content of the visual scene and visual attention of the speaker. Visual attention is used to focus on likely objects within the scene. Given a spoken description the system then uses the visually biased language model to process the speech. The system uses head pose as a proxy for the visual attention of the speaker. Readily available standard computer vision algorithms are used to recognize the objects in the scene and automatic real time head pose estimation is done using depth data captured via a Microsoft Kinect. The system was evaluated on multiple participants. Overall, incorporating visual information into the speech recognizer greatly improved speech recognition accuracy. The rapidly decreasing cost of 3D sensing technologies such as the Kinect allows systems with similar underlying principles to be used for many speech recognition tasks where there is visual information.
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Program in Media Arts and Sciences (Massachusetts Institute of Technology)
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/2556288.2556957
