Investigating the Effect of the Multiple Comparisons Problem in Visual Analysis
Name
zgraggeninvestigating.pdf
Description
Accepted version
Size
601.44 KB
Format
Adobe PDF
Checksum (MD5)
360dd66f98f396ab4b1a719d260e90e9
Author(s)
Zgraggen, Emanuel
Zhao, Zheguang
Zeleznik, Robert
Kraska, Tim
Date Issued
April 21, 2018
Journal
Conference on Human Factors in Computing Systems - Proceedings
Publisher
ACM
Citation
Zgraggen, Emanuel, Zhao, Zheguang, Zeleznik, Robert and Kraska, Tim. 2018. "Investigating the Effect of the Multiple Comparisons Problem in Visual Analysis." Conference on Human Factors in Computing Systems - Proceedings, 2018-April.
Version
Author's final manuscript
Abstract
© 2018 Association for Computing Machinery. The goal of a visualization system is to facilitate data-driven insight discovery. But what if the insights are spurious? Features or patterns in visualizations can be perceived as relevant insights, even though they may arise from noise. We often compare visualizations to a mental image of what we are interested in: a particular trend, distribution or an unusual pattern. As more visualizations are examined and more comparisons are made, the probability of discovering spurious insights increases. This problem is well-known in Statistics as the multiple comparisons problem (MCP) but overlooked in visual analysis. We present a way to evaluate MCP in visualization tools by measuring the accuracy of user reported insights on synthetic datasets with known ground truth labels. In our experiment, over 60% of user insights were false. We show how a confirmatory analysis approach that accounts for all visual comparisons, insights and non-insights, can achieve similar results as one that requires a validation dataset.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3173574.3174053
