Estimating Frequency Distributions in Data Streams

Abstract

Streaming algorithms allow for space-efficient processing of massive datasets. The distribution of the frequencies of items in a large dataset is often used to characterize that data: e.g., the data is heavy-tailed, the data follows a power law, or there are many elements that only appear only once or twice. In this thesis, we focus on the problem of estimating the profile (a vector representation of the frequency distribution). Given a sequence of m elements from a universe of size n, its profile is a vector φ whose i-th entry φᵢ represents the number of distinct elements that appear in the stream exactly i times. A classic paper by Datar and Muthukrishan from 2002 gave an algorithm which estimates any entry φᵢ up to an additive error of ±εD using O(1/ε² log(nm)) bits of space, where D is the number of distinct elements in the stream. We considerably improve on this result by designing an algorithm which estimates the whole profile vector φ, up to overall error ±εm, using O(1/ε2 log(1/ε) + log(nm)) bits. More formally, we give an algorithm that computes an approximate profile φˆ such that the L₁ distance ‖φ − φˆ‖₁ is at most εm. In addition to bounding the error across all coordinates, our space bound separates the terms that depend on 1/ε and those that depend on n and m. Furthermore, we give a lower bound showing that our bound is optimal up to constant factors.

To achieve these results, we introduce two new techniques. First, we develop hashing-based sketches that keep very limited information about the identities of the hashed elements. As a result, elements with different frequencies are mixed together, and need to be unmixed using an iterative “deconvolution” process. Second, we reduce the randomness used by the algorithms in a somewhat subtle way: we first use Nisans generator to ensure that the random variables of interest are O(1)-wise independent, and then we analyze those variables by calculating their moments. (In our setting, using Nisans generator alone would not yield the desired space bound.) The latter technique seems quite versatile, and has been already used for other streaming problems [Ano23].