2026-09-19T02:05:31Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1221012026-06-16T18:16:13Zcom_1721.1_33226com_1721.1_7582com_1721.1_7581col_1721.1_145728col_1721.1_131022
Antonio Torralba.
Zhao, Hang,Ph.D.Massachusetts Institute of Technology.
Massachusetts Institute of Technology. Department of Mechanical Engineering.
Massachusetts Institute of Technology. Department of Mechanical Engineering
2019-09-16T18:37:48Z
2019-09-16T18:37:48Z
2019
2019
https://hdl.handle.net/1721.1/122101
1117714214
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Thesis: Ph. D. in Mechanical Engineering and Computation, Massachusetts Institute of Technology, Department of Mechanical Engineering, 2019
Cataloged from student-submitted PDF version of thesis.
Includes bibliographical references (pages 121-132).
Scene parsing is a fundamental topic in computer vision and computational audition, where people develop computational approaches to achieve human perceptual system's ability in understanding scenes, e.g. group visual regions of an image into objects and segregate sound components in a noisy environment. This thesis investigates fully-supervised and self-supervised machine learning approaches to parse visual and auditory signals, including images, videos, and audios. Visual scene parsing refers to densely grouping and labeling of image regions into object concepts. First I build the MIT scene parsing benchmark based on a large scale, densely annotated dataset ADE20K. This benchmark, together with the state-of-the-art models we open source, offers a powerful tool for the research community to solve semantic and instance segmentation tasks. Then I investigate the challenge of parsing a large number of object categories in the wild. An open vocabulary scene parsing model which combines a convolutional neural network with a structured knowledge graph is proposed to address the challenge. Auditory scene parsing refers to recognizing and decomposing sound components in complex auditory environments. I propose a general audio-visual self-supervised learning framework that learns from a large amount of unlabeled internet videos. The learning process discovers the natural synchronization of vision and sounds without human annotation. The learned model achieves the capability to localize sound sources in videos and separate them from mixture. Furthermore, I demonstrate that motion cues in videos are tightly associated with sounds, which help in solving sound localization and separation problems.
by Hang Zhao.
Ph.D. in Mechanical Engineering and Computation
Ph.D.inMechanicalEngineeringandComputation Massachusetts Institute of Technology, Department of Mechanical Engineering
132 pages
eng
Massachusetts Institute of Technology
MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.
http://dspace.mit.edu/handle/1721.1/7582
Mechanical Engineering.
Visual and auditory scene parsing
Thesis
application/pdf
2019-09-16T18:37:45Z
Publication
Doctoral
MechE
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="cfe38f0e-c87a-4aad-97d9-cf170624d704">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Language>eng</Language>
<Title>Visual and auditory scene parsing</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2019</PublicationDate>
<Authors>
<Author>
<DisplayName>Zhao, Hang,Ph.D.Massachusetts Institute of Technology.</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://dspace.mit.edu/handle/1721.1/7582</License>
<Keyword>Mechanical Engineering.</Keyword>
<Abstract>Scene parsing is a fundamental topic in computer vision and computational audition, where people develop computational approaches to achieve human perceptual system's ability in understanding scenes, e.g. group visual regions of an image into objects and segregate sound components in a noisy environment. This thesis investigates fully-supervised and self-supervised machine learning approaches to parse visual and auditory signals, including images, videos, and audios. Visual scene parsing refers to densely grouping and labeling of image regions into object concepts. First I build the MIT scene parsing benchmark based on a large scale, densely annotated dataset ADE20K. This benchmark, together with the state-of-the-art models we open source, offers a powerful tool for the research community to solve semantic and instance segmentation tasks. Then I investigate the challenge of parsing a large number of object categories in the wild. An open vocabulary scene parsing model which combines a convolutional neural network with a structured knowledge graph is proposed to address the challenge. Auditory scene parsing refers to recognizing and decomposing sound components in complex auditory environments. I propose a general audio-visual self-supervised learning framework that learns from a large amount of unlabeled internet videos. The learning process discovers the natural synchronization of vision and sounds without human annotation. The learned model achieves the capability to localize sound sources in videos and separate them from mixture. Furthermore, I demonstrate that motion cues in videos are tightly associated with sounds, which help in solving sound localization and separation problems.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>