Behavior-driven optimization techniques for scalable data exploration

Abstract

Interactive visualizations are a popular medium used by scientists to explore, analyze and generally make sense of their data. However, with the overwhelming amounts of data that scientists collect from various instruments (e.g., telescopes, satellites, gene sequencers and field sensors), they need ways of efficiently transforming their data into interactive visualizations. Though a variety of visualization tools exist to help people make sense of their data, these tools often rely on database management systems (or DBMSs) for data processing and storage; and unfortunately, DBMSs fail to process the data fast enough to support a fluid, interactive visualization experience. This thesis blends optimization techniques from databases and methodology from HCI and visualization in order to support interactive and iterative exploration of large datasets. Our main goal is to reduce latency in visualization systems, i.e., the time these systems spend responding to a user's actions. We demonstrate through a comprehensive user study that latency has a clear (negative) effect on users' high-level analysis strategies, which becomes more pronounced as the latency is increased. Furthermore, we find that users are more susceptible to the effects of system latency when they have existing domain knowledge, a common scenario for data scientists. We then developed a visual exploration system called Sculpin that utilizes a suite of optimizations to reduce system latency. Sculpin learns user exploration patterns automatically, and exploits these patterns to pre-fetch data ahead of users as they explore. We then combine data-prefetching with incremental data processing (i.e., incremental materialization) and visualization-focused caching optimizations to further boost performance. With all three of these techniques (pre-fetching, caching, and pre-computation), Sculpin is able to: create visualizations 380% faster and respond to user interactions 88% faster than existing visualization systems, while also using less than one third of the space required by other systems to store materialized query results.