SubZero: A fine-grained lineage system for scientific databases
Author(s)Wu, Eugene; Madden, Samuel R.; Stonebraker, Michael
MetadataShow full item record
Data lineage is a key component of provenance that helps scientists track and query relationships between input and output data. While current systems readily support lineage relationships at the file or data array level, finer-grained support at an array-cell level is impractical due to the lack of support for user defined operators and the high runtime and storage overhead to store such lineage. We interviewed scientists in several domains to identify a set of common semantics that can be leveraged to efficiently store fine-grained lineage. We use the insights to define lineage representations that efficiently capture common locality properties in the lineage data, and a set of APIs so operator developers can easily export lineage information from user defined operators. Finally, we introduce two benchmarks derived from astronomy and genomics, and show that our techniques can reduce lineage query costs by up to 10× while incuring substantially less impact on workflow runtime and storage.
DepartmentMassachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Proceedings of the 2013 IEEE 29th International Conference on Data Engineering (ICDE)
Institute of Electrical and Electronics Engineers (IEEE)
Wu, Eugene, Samuel Madden, and Michael Stonebraker. “SubZero: A Fine-Grained Lineage System for Scientific Databases.” 2013 IEEE 29th International Conference on Data Engineering (ICDE) (April 8-12, 2013). Brisbane, QLD. IEEE. p.865-876.
Author's final manuscript
INSPEC Accession Number: 13598422