Extracting Syntactical Patterns from Databases
Name
1710.11528.pdf
Description
Submitted version
Size
444.25 KB
Format
Adobe PDF
Checksum (MD5)
fcf92ae86ba9e0dc532cb7d90b0054bb
Author(s)
Ilyas, Andrew
M. F. da Trindade, Joana
Castro Fernandez, Raul
Madden, Samuel
Date Issued
April 2018
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Ilyas, Andrew, M. F. da Trindade, Joana, Castro Fernandez, Raul and Madden, Samuel. 2018. "Extracting Syntactical Patterns from Databases."
Version
Original manuscript
Abstract
© 2018 IEEE. Many database columns contain string or numerical data that conforms to a pattern, such as phone numbers, dates, addresses, product identifiers, and employee ids. These patterns are useful in a number of data processing applications, including understanding what a specific field represents when field names are ambiguous, identifying outlier values, and finding similar fields across data sets.One way to express such patterns would be to learn regular expressions for each field in the database. Unfortunately, existing techniques on regular expression learning are slow, taking hundreds of seconds for columns of just a few thousand values. In contrast, we develop XSYSTEM, an efficient method to learn patterns over database columns in significantly less time.We show that these patterns can not only be built quickly, but are expressive enough to capture a number of key applications, including detecting outliers, measuring column similarity, and assigning semantic labels to columns (based on a library of regular expressions). We evaluate these applications with datasets that range from chemical databases (based on a collaboration with a pharmaceutical company), our university data warehouse, and open data from MassData.gov.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/icde.2018.00014
