Work-sharing framework for Apache Spark

Abstract

Apache Spark is a popular framework for distributed data processing that generalizes the MapReduce model and significantly improves the performance of many use cases. People can use Spark to query enormous data sets faster than before to gain insights for a competitive edge in industry. Often these ad-hoc queries perform similar work, and there is an opportunity to share the work of different queries. This can reduce the total computation time even more. We have developed a Wrapper class which performs such optimizations. In particular, its strategy of lazy evaluation allows duplicate computation to be avoided and multiple related Spark jobs to be executed at the same time, reducing the scheduling overhead. Overall, the system demonstrates significant efficiency gains when compared to default Spark.