On mitigating the analytical limitations of finely stratified experiments
Name
1706.06469.pdf
Size
326.01 KB
Format
Adobe PDF
Checksum (MD5)
bf3f163f2ec0b28a134f8118c0a18fab
Author(s)
Fogarty, Colin B
Date Issued
August 2018
Journal
Journal of the Royal Statistical Society: Series B (Statistical Methodology)
Publisher
Royal Statistical Society
Citation
Fogarty, Colin B. “On Mitigating the Analytical Limitations of Finely Stratified Experiments.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 80, no. 5 (August 13, 2018): 1035–1056.
Version
Original manuscript
Abstract
Although attractive from a theoretical perspective, finely stratified experiments such as paired designs suffer from certain analytical limitations that are not present in block-randomized experiments with multiple treated and control individuals in each block. In short, when using a weighted difference in means to estimate the sample average treatment effect, the traditional variance estimator in a paired experiment is conservative unless the pairwise average treatment effects are constant across pairs; however, in more coarsely stratified experiments, the corresponding variance estimator is unbiased if treatment effects are constant within blocks, even if they vary across blocks. Using insights from classical least squares theory, we present an improved variance estimator that is appropriate in finely stratified experiments. The variance estimator remains conservative in expectation but is asymptotically no more conservative than the classical estimator and can be considerably less conservative. The magnitude of the improvement depends on the extent to which effect heterogeneity can be explained by observed covariates. Aided by this estimator, a new test for the null hypothesis of a constant treatment effect is proposed. These findings extend to some, but not all, superpopulation models, depending on whether the covariates are viewed as fixed across samples.
MIT Department
Sloan School of Management
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1111/rssb.12290
