Outlier Detection in Heterogeneous Datasets using Automatic Tuple Expansion
Name
MIT-CSAIL-TR-2016-002.pdf
Size
2.58 MB
Format
Adobe PDF
Checksum (MD5)
627837040cade5057d5031b427024c45
Download all files submitted through automated deposit
Author(s)
Pit-Claudel, Clément
Mariet, Zelda
Harding, Rachael
Madden, Sam
Advisor(s)
Adam Chlipala
Date Issued
February 8, 2016
Series/Report no.
MIT-CSAIL-TR-2016-002
Abstract
Rapidly developing areas of information technology are generating massive amounts of data. Human errors, sensor failures, and other unforeseen circumstances unfortunately tend to undermine the quality and consistency of these datasets by introducing outliers -- data points that exhibit surprising behavior when compared to the rest of the data. Characterizing, locating, and in some cases eliminating these outliers offers interesting insight about the data under scrutiny and reinforces the confidence that one may have in conclusions drawn from otherwise noisy datasets. In this paper, we describe a tuple expansion procedure which reconstructs rich information from semantically poor SQL data types such as strings, integers, and floating point numbers. We then use this procedure as the foundation of a new user-guided outlier detection framework, dBoost, which relies on inference and statistical modeling of heterogeneous data to flag suspicious fields in database tuples. We show that this novel approach achieves good classification performance, both in traditional numerical datasets and in highly non-numerical contexts such as mostly textual datasets. Our implementation is publicly available, under version 3 of the GNU General Public License.
Persistent DSpace Link
