Near-Optimal A-B Testing
Name
experiment_design.pdf
Description
Submitted version
Size
1.43 MB
Format
Adobe PDF
Checksum (MD5)
9a125f0a8a2902f7947e4047139a9b61
Author(s)
Bhat, Nikhil
Farias, Vivek F
Moallemi, Ciamac C
Sinha, Deeksha
Date Issued
2020
Journal
Management Science
Publisher
Institute for Operations Research and the Management Sciences (INFORMS)
Version
Original manuscript
Abstract
We consider the problem of A-B testing when the impact of the treatment is marred by a large number of covariates. Randomization can be highly inefficient in such settings, and thus we consider the problem of optimally allocating test subjects to either treatment with a view to maximizing the precision of our estimate of the treatment effect. Our main contribution is a tractable algorithm for this problem in the online setting, where subjects arrive, and must be assigned, sequentially, with covariates drawn from an elliptical distribution with finite second moment. We further characterize the gain in precision afforded by optimized allocations relative to randomized allocations, and show that this gain grows large as the number of covariates grows. Our dynamic optimization framework admits several generalizations that incorporate important operational constraints such as the consideration of selection bias, budgets on allocations, and endogenous stopping times. In a set of numerical experiments, we demonstrate that our method simultaneously offers better statistical efficiency and less selection bias than state-of-the-art competing biased coin designs.
MIT Department
Sloan School of Management
Massachusetts Institute of Technology. Operations Research Center
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1287/MNSC.2019.3424
