Distributed dynamic partially stateful dataflow

Abstract

This thesis present a distributed implementation of Noria, a new streaming dataflow system that simplifies the infrastructure of read-heavy web applications by unifying the database, caching layer, and parts of application logic in a single system. Noria's partially-stateful dataflow allows it to evict and reconstruct state on demand, and avoid prior dataflow systems' restriction to windowed state. Unlike existing dataflow systems, Noria adapts on-line to schema and query changes, and shares state and computation across related queries to eliminate duplicate effort. Noria's distributed design enables it to leverage the compute power of an entire cluster while providing high availability thanks to its fault tolerant design. On a single machine, Noria already outperforms MySQL by up to 7 x, but when running across a cluster of machines, it can scale to tens of millions of reads and millions of writes per second.