Applying sampling and predicate pushdown in an interactive data exploration system

Abstract

Interactive data exploration (IDE) systems require low latency and high performance, as users expect to see their ad-hoc queries return results quickly for a seamless experience. Predicate pushdown is a common performance optimization for systems that rely on databases, by pushing the filtering that was originally performed by the overarching system down to its underlying database systems. In this work, we implement both sampling and predicate pushdown in Northstar, a system for interactive data science. We then investigate and benchmark optimization strategies for predicate pushdown in Northstar, and find that a cache-aware "adaptive" pushdown strategy leads in the greatest performance gain in many cases.