Efficient Versioning for Scientific Array Databases
Author(s)Seering, Adam; Cudre-Mauroux, Philippe; Stonebraker, Michael; Madden, Samuel R.
MetadataShow full item record
In this paper, we describe a versioned database storage manager we are developing for the SciDB scientific database. The system is designed to efficiently store and retrieve array-oriented data, exposing a "no-overwrite" storage model in which each update creates a new "version" of an array. This makes it possible to perform comparisons of versions produced at different times or by different algorithms, and to create complex chains and trees of versions. We present algorithms to efficiently encode these versions, minimizing storage space while still providing efficient access to the data. Additionally, we present an optimal algorithm that, given a long sequence of versions, determines which versions to encode in terms of each other (using delta compression) to minimize total storage space or query execution cost. We compare the performance of these algorithms on real world data sets from the National Oceanic and Atmospheric Administration (NOAA), Open Street Maps, and several other sources. We show that our algorithms provide better performance than existing version control systems not optimized for array data, both in terms of storage size and access time, and that our delta-compression algorithms are able to substantially reduce the total storage space when versions exist with a high degree of similarity.
DepartmentMassachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Proceedings of the 2012 IEEE 28th International Conference on Data Engineering
Institute of Electrical and Electronics Engineers (IEEE)
Seering, Adam, Philippe Cudre-Mauroux, Samuel Madden, and Michael Stonebraker. “Efficient Versioning for Scientific Array Databases.” 2012 IEEE 28th International Conference on Data Engineering (April 2012).
Author's final manuscript