Show simple item record

dc.contributor.authorYeh, Tom
dc.contributor.authorChang, Tsung-Hsiang
dc.contributor.authorMiller, Robert C.
dc.date.accessioned2012-09-13T14:57:06Z
dc.date.available2012-09-13T14:57:06Z
dc.date.issued2009-10
dc.identifier.isbn978-1-60558-745-5
dc.identifier.urihttp://hdl.handle.net/1721.1/72686
dc.description.abstractWe present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.en_US
dc.language.isoen_US
dc.publisherAssociation for Computing Machinery (ACM)en_US
dc.relation.isversionofhttp://dx.doi.org/10.1145/1622176.1622213en_US
dc.rightsCreative Commons Attribution-Noncommercial-Share Alike 3.0en_US
dc.rights.urihttp://creativecommons.org/licenses/by-nc-sa/3.0/en_US
dc.sourceOther University Web Domainen_US
dc.titleSikuli: Using GUI screenshots for search and automationen_US
dc.typeArticleen_US
dc.identifier.citationTom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). ACM, New York, NY, USA, 183-192.en_US
dc.contributor.departmentMassachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratoryen_US
dc.contributor.departmentMassachusetts Institute of Technology. Department of Electrical Engineering and Computer Scienceen_US
dc.contributor.approverMiller, Robert C.
dc.contributor.mitauthorYeh, Tom
dc.contributor.mitauthorChang, Tsung-Hsiang
dc.contributor.mitauthorMiller, Robert C.
dc.relation.journalProceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09)en_US
dc.eprint.versionAuthor's final manuscripten_US
dc.type.urihttp://purl.org/eprint/type/ConferencePaperen_US
dspace.orderedauthorsYeh, Tom; Chang, Tsung-Hsiang; Miller, Robert C.en
dc.identifier.orcidhttps://orcid.org/0000-0002-0442-691X
mit.licenseOPEN_ACCESS_POLICYen_US
mit.metadata.statusComplete


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record