This is not the latest version of this item. The latest version can be found here.
Pattern functional dependencies for data cleaning
Name
3377369.3377377.pdf
Description
Published version
Size
593.64 KB
Format
Adobe PDF
Checksum (MD5)
7d7e5106df59200157230016e7236387
Author(s)
Qahtan, Abdulhakim
Tang, Nan
Ouzzani, Mourad
Cao, Yang
Stonebraker, Michael
Date Issued
2020
Journal
Proceedings of the VLDB Endowment
Publisher
VLDB Endowment
Version
Final published version
Abstract
Patterns (or regex-based expressions) are widely used to constrain the format of a domain (or a column), e.g., a Year column should contain only four digits, and thus a value like “1980-“ might be a typo. Moreover, integrity constraints (ICs) defined over multiple columns, such as (conditional) functional dependencies and denial constraints, e.g., a ZIP code uniquely determines a city in the UK, have been widely used in data cleaning. However, a promising, but not yet explored, direction is to combine regex- and IC-based theories to capture data dependencies involving partial attribute values. For example, in an employee ID such as“F-9-107“, “F“ is sufficient to determine the finance department. Inspired by the above observation, we propose a novel class of ICs, called pattern functional dependencies (PFDs), to model fine-grained data dependencies gleaned from partial attribute values. These dependencies cannot be modeled using traditional ICs, such as (conditional) functional dependencies, which work on entire attribute values. We also present a set of axioms for the inference of PFDs, analogous to Armstrong's axioms for FDs, and study the complexity of consistency and implication analysis of PFDs. Moreover, we devise an effective algorithm to automatically discover PFDs even in the presence of errors in the data. Our extensive experiments on 15 real-world datasets show that our approach can effectively discover valid and useful PFDs over dirty data, which can then be used to detect data errors that are hard to capture by other types of ICs.
Terms of Use
Creative Commons Attribution-NonCommercial-NoDerivs License
Persistent DSpace Link
DOI of Published Version
10.14778/3377369.3377377
