Estimating entropy of distributions in constant space
MetadataShow full item record
We consider the task of estimating the entropy of k-ary distributions from samples in the streaming model, where space is limited. Our main contribution is an algorithm that requires O ( klog(1"3/")2 ) samples and a constant O(1) memory words of space and outputs a ±" estimate of H(p). Without space limitations, the sample complexity has been established as S(k, ") = T ( "logkk + log"22 k 0, which is sub-linear in the domain size k, and the current algorithms that achieve optimal sample complexity also require nearly-linear space in k. Our algorithm partitions [0, 1] into intervals and estimates the entropy contribution of probability values in each interval. The intervals are designed to trade off the bias and variance of these estimates.
DepartmentMassachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Advances in Neural Information Processing Systems
Neural Information Processing Systems Foundation
Acharya, Jayadev et al. “Estimating entropy of distributions in constant space.” Advances in Neural Information Processing Systems (NeurIPS 2019), December 2019, Vancouver, Canada, Neural Information Processing Systems Foundation, December 2019. © 2019 The Author(s)
Final published version