Behavior-driven optimization techniques for scalable data exploration
Author(s)
Battle, Leilani Marie
DownloadFull printable version (10.16Mb)
Other Contributors
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.
Advisor
Michael Stonebraker.
Terms of use
Metadata
Show full item recordAbstract
Interactive visualizations are a popular medium used by scientists to explore, analyze and generally make sense of their data. However, with the overwhelming amounts of data that scientists collect from various instruments (e.g., telescopes, satellites, gene sequencers and field sensors), they need ways of efficiently transforming their data into interactive visualizations. Though a variety of visualization tools exist to help people make sense of their data, these tools often rely on database management systems (or DBMSs) for data processing and storage; and unfortunately, DBMSs fail to process the data fast enough to support a fluid, interactive visualization experience. This thesis blends optimization techniques from databases and methodology from HCI and visualization in order to support interactive and iterative exploration of large datasets. Our main goal is to reduce latency in visualization systems, i.e., the time these systems spend responding to a user's actions. We demonstrate through a comprehensive user study that latency has a clear (negative) effect on users' high-level analysis strategies, which becomes more pronounced as the latency is increased. Furthermore, we find that users are more susceptible to the effects of system latency when they have existing domain knowledge, a common scenario for data scientists. We then developed a visual exploration system called Sculpin that utilizes a suite of optimizations to reduce system latency. Sculpin learns user exploration patterns automatically, and exploits these patterns to pre-fetch data ahead of users as they explore. We then combine data-prefetching with incremental data processing (i.e., incremental materialization) and visualization-focused caching optimizations to further boost performance. With all three of these techniques (pre-fetching, caching, and pre-computation), Sculpin is able to: create visualizations 380% faster and respond to user interactions 88% faster than existing visualization systems, while also using less than one third of the space required by other systems to store materialized query results.
Description
Thesis: Ph. D., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2017. This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Cataloged from student-submitted PDF version of thesis. Includes bibliographical references (pages 153-162).
Date issued
2017Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer SciencePublisher
Massachusetts Institute of Technology
Keywords
Electrical Engineering and Computer Science.