Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. MIT Open Access Articles
  3. MIT Open Access Articles
  4. Optimal approximate sampling from discrete probability distributions

Optimal approximate sampling from discrete probability distributions

Thumbnail Image
Download
Name

3371104.pdf

Description
Published version
Size

9.26 MB

Format

Adobe PDF

Checksum (MD5)

b00fb6529bf10768896e9174d6a278c3

sword-2021-03-22T15:19:17.original.xml (130 B)
Original SWORD entry document
Author(s)
Saad, Feras A
•
Freer, Cameron E
•
Rinard, Martin C
•
Mansinghka, Vikash K
Date Issued
2020
Journal
Proceedings of the ACM on Programming Languages
Publisher
Association for Computing Machinery (ACM)
Version
Final published version
Abstract
© 2020 Copyright held by the owner/author(s). This paper addresses a fundamental problem in random variate generation: given access to a random source that emits a stream of independent fair bits, what is the most accurate and entropy-efficient algorithm for sampling from a discrete probability distribution (p1, . . . ,pn), where the probabilities of the output distribution (p'1, . . . ,p'n) of the sampling algorithm must be specified using at most k bits of precision? We present a theoretical framework for formulating this problem and provide new techniques for finding sampling algorithms that are optimal both statistically (in the sense of sampling accuracy) and information-theoretically (in the sense of entropy consumption). We leverage these results to build a system that, for a broad family of measures of statistical accuracy, delivers a sampling algorithm whose expected entropy usage is minimal among those that induce the same distribution (i.e., is "entropy-optimal") and whose output distribution (p'1, . . . ,p'n) is a closest approximation to the target distribution (p1, . . . ,pn) among all entropy-optimal sampling algorithms that operate within the specified k-bit precision. This optimal approximate sampler is also a closer approximation than any (possibly entropy-suboptimal) sampler that consumes a bounded amount of entropy with the specified precision, a class which includes floating-point implementations of inversion sampling and related methods found in many software libraries. We evaluate the accuracy, entropy consumption, precision requirements, and wall-clock runtime of our optimal approximate sampling algorithms on a broad set of distributions, demonstrating the ways that they are superior to existing approximate samplers and establishing that they often consume significantly fewer resources than are needed by exact samplers.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
Creative Commons Attribution NonCommercial License 4.0
https://creativecommons.org/licenses/by-nc/4.0/
Persistent DSpace Link
https://hdl.handle.net/1721.1/135919
DOI of Published Version
https://doi.org/10.1145/3371104
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.