Efficient Versioning for Scientific Array Databases
Name
Madden_Efficient versioning.pdf
Size
687.52 KB
Format
Adobe PDF
Checksum (MD5)
f328bfbbd8b24e7078c1f582652e06a5
Author(s)
Seering, Adam
Cudre-Mauroux, Philippe
Stonebraker, Michael
Madden, Samuel R.
Date Issued
April 2012
Journal
Proceedings of the 2012 IEEE 28th International Conference on Data Engineering
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Seering, Adam, Philippe Cudre-Mauroux, Samuel Madden, and Michael Stonebraker. “Efficient Versioning for Scientific Array Databases.” 2012 IEEE 28th International Conference on Data Engineering (April 2012).
Version
Author's final manuscript
Abstract
In this paper, we describe a versioned database storage manager we are developing for the SciDB scientific database. The system is designed to efficiently store and retrieve array-oriented data, exposing a "no-overwrite" storage model in which each update creates a new "version" of an array. This makes it possible to perform comparisons of versions produced at different times or by different algorithms, and to create complex chains and trees of versions. We present algorithms to efficiently encode these versions, minimizing storage space while still providing efficient access to the data. Additionally, we present an optimal algorithm that, given a long sequence of versions, determines which versions to encode in terms of each other (using delta compression) to minimize total storage space or query execution cost. We compare the performance of these algorithms on real world data sets from the National Oceanic and Atmospheric Administration (NOAA), Open Street Maps, and several other sources. We show that our algorithms provide better performance than existing version control systems not optimized for array data, both in terms of storage size and access time, and that our delta-compression algorithms are able to substantially reduce the total storage space when versions exist with a high degree of similarity.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/ICDE.2012.102
