Understanding sketch-and-speech descriptions of machines

Abstract

In this thesis, we present PhysInk, a sketching application that provides a natural way to describe physical structure and behavior to a computer. The user describes structure by sketching physical objects and constraints on a canvas backed by a 2D physics engine. The user can then move the objects to explicitly demonstrate physical behavior and can utter simple speech commands to label objects and events. These interactions are captured in PhysInk's timeline - a causal graph of physical events - which is used to build a deeper understanding of behavior. The events capture the geometry of movement and contact between objects, while the timeline captures the causal relationships between events. Capturing behavior in this framework enables a more intelligent conversation with the user about function. First, it enables assisted editing, where PhysInk propagates the user's edits of behavior through the timeline in order to maintain a logical sequence of events at all times. Second, it allows users to produce a physically-realistic simulation of the behavior that they have described. These features demonstrate the usefulness and depth that the timeline offers as a knowledge representation for behavior, making PhysInk the first in a new class of design tools that focuses on function, as well as form.