Remote data access and analysis using SciDB

Abstract

SciDB is an innovative data analysis system that provides fast querying and manipulation of large amounts of time-series, scientific data. This thesis describes the design of a framework that provides a user interface to SciDB that facilitates interactive processing of large datasets and supports long-running batch jobs on a remote server or cluster. Using this interface, python user scripts access SciDB data, process it, and write new result arrays. The framework addresses problems such as garbage collection, data access permissions and maintenance of provenance. We present a case study in which we apply this framework to data from the WaterWiSe project, and analyze the runtime performance of the system.