Associating the visual representation of user interfaces with their internal structures and metadata
Author(s)Chang, Tsung-Hsiang; Yeh, Tom; Miller, Robert C.
MetadataShow full item record
Pixel-based methods are emerging as a new and promising way to develop new interaction techniques on top of existing user interfaces. However, in order to maintain platform independence, other available low-level information about GUI widgets, such as accessibility metadata, was neglected intentionally. In this paper, we present a hybrid framework, PAX, which associates the visual representation of user interfaces (i.e. the pixels) and their internal hierarchical metadata (i.e. the content, role, and value). We identify challenges to building such a framework. We also develop and evaluate two new algorithms for detecting text at arbitrary places on the screen, and for segmenting a text image into individual word blobs. Finally, we validate our framework in implementations of three applications. We enhance an existing pixel-based system, Sikuli Script, and preserve the readability of its script code at the same time. Further, we create two novel applications, Screen Search and Screen Copy, to demonstrate how PAX can be applied to development of desktop-level interactive systems.
DepartmentMassachusetts Institute of Technology. Department of Electrical Engineering and Computer Science; Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Proceedings of the 24th annual ACM symposium on User interface software and technology
Association for Computing Machinery (ACM)
Chang, Tsung-Hsiang, Tom Yeh and Rob Miller. "Associating the visual representation of user interfaces with their internal structures and metadata." In UIST'11, Proceedings of the 24th annual ACM symposium on User interface software and technology, Santa Barbara, California, USA, October 16-19, 2011, pages 245-256.
Author's final manuscript