SparkSim: A Counterfactual Approach for Spark Cluster Scheduling

Abstract

Simulating and testing scheduling policies can be immensely time- and resourceintensive. In this work, we explore a novel approach, SparkSim, to scheduling policy training that is faster and more efficient than traditional scheduling policy testing. Our approach is based on an extension of CausalSim’s existing trace-driven approach [3], which we apply to replace the current Spark Cluster scheduling policy testing in simulation.

To simulate the runtime under a new scheduling policy, our method consists of training a neural model to learn about unseen and unbiased computation elements of the cluster, extracting them, and using them as latents in predicting the duration of a workload from an existing trace. We implement this using a counterfactual approach, which takes a trace that was executed to predict a new one as if it had taken place under the same cluster conditions.

My thesis focuses on evaluating and investigating the performance of SparkSim. We evaluate SparkSim on two baselines that do not require training. Our results show that SparkSim underperforms against these baselines during easier prediction tasks (such as copying from source), but outperforms them when the prediction tasks get harder. Future work lends itself to greatly improve upon these results.