Drawing on the World: sketch in context

Abstract

This thesis introduces the idea that combining sketch recognition with contextual data-information about what is being drawn on-can improve the recognition of meaning in sketch and enrich the user interaction experience. I created a language called StepStool that facilitates the description of the relationship between digital ink and contextual data, and wrote the corresponding interpreter that enables my system to distinguish between gestural commands issued to an autonomous forklift. A user study was done to compare the correctness of a sketch interface with and without context on the canvas. This thesis coins the phrase "Drawing on the World" to mean contextual sketch recognition, describes the implementation and methodology behind "Drawing on the World", describes the forklift's interface, and discusses other possible uses for a contextual gesture recognizer. Sample code is provided that describes the specifics of the StepStool engine's implementation and the implementation of the forklift's interface.