Insight : interactive machine learning for complex graphics selection

Abstract

Modern vector graphics editors support the creation of a wonderful variety of complex designs and artwork. Users produce highly realistic illustrations, stylized representational art, even nuanced data visualizations. In light of these complex graphics, selections, representations of sets of objects that users want to manipulate, become more complex as well. Direct manipulation tools that artists and designers find accessible and useful for editing graphics such as logos and icons do not have the same applicability in these more complex cases. Given that selection is the first step for nearly all editing in graphics, it is important to enable artists and designers to express these complex selections. This thesis explores the use of interactive machine learning techniques to improve direct selection interfaces. To investigate this approach, I created Insight, an interactive machine learning selection tool for making a relevant class of complex selections: visually similar objects. To make a selection, users iteratively provide examples of selection objects by clicking on them in the graphic. Insight infers a selection from the examples at each step, allowing users to quickly understand results of actions and reactively shape the complex selection. The interaction resembles the direct manipulation interactions artists and designers have found accessible, while helping express complex selections by inferring many parameter changes from simple actions. I evaluated Insight in a user study of digital designers and artists, finding that Insight enabled users to effectively and easily make complex selections not supported by state-of-the-art vector graphics editors. My results contribute to existing work by both indicating a useful approach for providing complex representation access to artists and designers, and showing a new application for interactive machine learning.