2026-09-18T20:09:42Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1073522026-06-17T14:47:36Zcom_1721.1_7582com_1721.1_7581col_1721.1_131022

William Freeman. Owens, Andrew (Andrew Hale) Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2017-03-10T15:06:40Z 2017-03-10T15:06:40Z 2016 2016 http://hdl.handle.net/1721.1/107352 973329840 Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2016. Cataloged from PDF version of thesis. Includes bibliographical references (pages 93-104). From the clink of a mug placed onto a saucer to the bustle of a busy café, our days are filled with visual experiences that are accompanied by distinctive sounds. In this thesis, we show that these sounds can provide a rich training signal for learning visual models. First, we propose the task of predicting the sound that an object makes when struck as a way of studying physical interactions within a visual scene. We demonstrate this idea by training an algorithm to produce plausible soundtracks for videos in which people hit and scratch objects with a drumstick. Then, with human studies and automated evaluations on recognition tasks, we verify that the sounds produced by the algorithm convey information about actions and material properties. Second, we show that ambient audio - e.g., crashing waves, people speaking in a crowd - can also be used to learn visual models. We train a convolutional neural network to predict a statistical summary of the sounds that occur within a scene, and we demonstrate that the visual representation learned by the model conveys information about objects and scenes. by Andrew Owens. Ph.D. 104 pages eng Massachusetts Institute of Technology MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582 Electrical Engineering and Computer Science. Learning visual models from paired audio-visual examples Thesis application/pdf false Publication unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="d9de862a-26f6-4635-b6de-a2236acf6d81"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>eng</Language> <Title>Learning visual models from paired audio-visual examples</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2016</PublicationDate> <Authors> <Author> <DisplayName>Owens, Andrew (Andrew Hale)</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>http://dspace.mit.edu/handle/1721.1/7582</License> <Keyword>Electrical Engineering and Computer Science.</Keyword> <Abstract>From the clink of a mug placed onto a saucer to the bustle of a busy café, our days are filled with visual experiences that are accompanied by distinctive sounds. In this thesis, we show that these sounds can provide a rich training signal for learning visual models. First, we propose the task of predicting the sound that an object makes when struck as a way of studying physical interactions within a visual scene. We demonstrate this idea by training an algorithm to produce plausible soundtracks for videos in which people hit and scratch objects with a drumstick. Then, with human studies and automated evaluations on recognition tasks, we verify that the sounds produced by the algorithm convey information about actions and material properties. Second, we show that ambient audio - e.g., crashing waves, people speaking in a crowd - can also be used to learn visual models. We train a convolutional neural network to predict a statistical summary of the sounds that occur within a scene, and we demonstrate that the visual representation learned by the model conveys information about objects and scenes.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>