Using graphical representation of user interfaces as visual references

Thumbnail Image
Download
Name

818202064-MIT.pdf

Description
Full printable version
Size

17.66 MB

Format

Adobe PDF

Checksum (MD5)

b064c7f2056fa2b172207e3f9d9268fb

Author(s)
Zhang, Zongxiang
Advisor(s)
Rob Miller.
Date Issued
2012
Publisher
Massachusetts Institute of Technology
Abstract
My thesis investigates using a graphical representation of user interfaces - screenshots - as a direct visual reference to support various kinds of applications. We have built several systems to demonstrate and validate this idea in domains like searching documentation, GUI automation and testing, and cross-device information migration. In particular, Sikuli Search enables users to search documentation using screenshots of GUI elements instead of keywords. Sikuli Script enables users to programmatically control GUIs without support from the underlying applications. Sikuli Test lets GUI developers and testers create test scripts without coding. Deep Shot introduces a framework and interaction techniques to migrate work states across heterogeneous devices in one action, taking a picture. We also discuss challenges inherent in screenshot-based interactions and propose potential solutions and directions of future research.
Description
Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2012.
Cataloged from PDF version of thesis.
Includes bibliographical references (p. 129-133).
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
M.I.T. theses are protected by
copyright. They may be viewed from this source for any purpose, but
reproduction or distribution in any format is prohibited without written
permission. See provided URL for inquiries about permission.
Persistent DSpace Link
PrivacyPermissionsAccessibilityContact us
Notify us about copyright concerns.