Computational perception for multi-modal document understanding
Name
1084273965-MIT.pdf
Description
Full printable version
Size
64.44 MB
Format
Adobe PDF
Checksum (MD5)
d13742fae9689c6f20746aa1af324b77
Author(s)
Bylinskii, Zoya
Advisor(s)
Fredo Durand and Aude Oliva.
Date Issued
2018
Publisher
Massachusetts Institute of Technology
Abstract
Multimodal documents occur in a variety of forms, as graphs in technical reports, diagrams in textbooks, and graphic designs in bulletins. Humans can efficiently process the visual and textual information contained within to make decisions on topics including business, healthcare, and science. Building the computational tools to understand multimodal documents can have important applications for web search, information retrieval, captioning and summarization, and automated design. This thesis makes contributions on two fronts: (i) to the development of data collection methods for measuring how humans perceive multimodal documents (i.e., where they look, what they find important), and (ii) to the development of computer vision tools for automatically parsing and making predictions about multimodal documents (i.e., the subject matter they are about). Specifically, the crowdsourced attention data captured from our novel user interfaces is used to train neural network models to predict where people look in graphic designs and information visualizations, with demonstrated applications to thumbnailing, design retargeting, and interactive feedback within graphic design tools. Separately, our models for detecting visual elements and parsing text elements in infographics (information graphics) are used for topic prediction and to present a system for automatic summarization. This thesis makes contributions at the interface of human and computer vision, with applications to human-computer interfaces and design.
Description
Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2018.
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Cataloged from student-submitted PDF version of thesis.
Includes bibliographical references (pages 171-192).
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission.
Persistent DSpace Link
