Towards DIVE : A platform for automatic visualization and analysis of structured datasets

Abstract

Our world is filled with data describing complex systems like international trade, personal mobility, particle interactions, and genomes. To make sense of these systems, analysts often use visualization and statistical analysis to identify trends, patterns, or anomalies in their data. However, currently available software tools for visualizing and analyzing data have major limitations because they have prohibitively steep learning curves, often need domain-specific customization, and require users to know a priori what they want to see before they see it. Here, I present a new platform for exploratory data visualization and analysis that automatically presents users with inferred visualizations and analyses. By turning data visualization and analysis into an act of curation and selection, this platform aspires to democratize the techniques needed to understand and communicate data. Conceptually, for any dataset, there are a finite number of combinations of its elements. Therefore there are a finite number of common visualizations or analyses of a dataset. In other words, it should be possible to enumerate the whole space of possible visualizations and analyses for a given dataset. Analysts can then explore this space and select interesting results. To capture this intuition, we developed a conceptual framework inspired by set theory and relational algebra, and drawing upon existing work in visualization and database architecture. With these analytical abstractions, we rigorously characterize datasets, infer data models, enumerate visualizable or analyzable data structures, and score these data structures. We implement this framework in the Data Integration and Visualization Engine (DIVE), a web-based platform for anyone to efficiently analyze or visualize arbitrary structured datasets, and then to export and share their results. DIVE has been under development since March 2014 and will continue being development in order to have a alpha version available by September 2015 and a beta version by the end of 2015.