Sikuli: Using GUI screenshots for search and automation
Name
Miller_Sikuli.pdf
Size
1.16 MB
Format
Adobe PDF
Checksum (MD5)
330fccc4ca83fa89c0eba1aef695de25
Author(s)
Yeh, Tom
Chang, Tsung-Hsiang
Miller, Robert C.
Date Issued
October 2009
Journal
Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09)
Publisher
Association for Computing Machinery (ACM)
Citation
Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09). ACM, New York, NY, USA, 183-192.
Version
Author's final manuscript
Abstract
We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike 3.0
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/1622176.1622213
