SubZero: A fine-grained lineage system for scientific databases
Name
Stonebraker_Subzero.pdf
Size
457.87 KB
Format
Adobe PDF
Checksum (MD5)
3d985352cdc073d1acd31b706e83f844
Author(s)
Wu, Eugene
Madden, Samuel R.
Stonebraker, Michael
Date Issued
April 2013
Journal
Proceedings of the 2013 IEEE 29th International Conference on Data Engineering (ICDE)
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Wu, Eugene, Samuel Madden, and Michael Stonebraker. “SubZero: A Fine-Grained Lineage System for Scientific Databases.” 2013 IEEE 29th International Conference on Data Engineering (ICDE) (April 8-12, 2013). Brisbane, QLD. IEEE. p.865-876.
Version
Author's final manuscript
Abstract
Data lineage is a key component of provenance that helps scientists track and query relationships between input and output data. While current systems readily support lineage relationships at the file or data array level, finer-grained support at an array-cell level is impractical due to the lack of support for user defined operators and the high runtime and storage overhead to store such lineage. We interviewed scientists in several domains to identify a set of common semantics that can be leveraged to efficiently store fine-grained lineage. We use the insights to define lineage representations that efficiently capture common locality properties in the lineage data, and a set of APIs so operator developers can easily export lineage information from user defined operators. Finally, we introduce two benchmarks derived from astronomy and genomics, and show that our techniques can reduce lineage query costs by up to 10× while incuring substantially less impact on workflow runtime and storage.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/ICDE.2013.6544881
