Promises and Perils of Pre-Analysis Plans
Name
Olken_Promises and perils.pdf
Size
664.83 KB
Format
Adobe PDF
Checksum (MD5)
5b598d7d6ac675507d488fbdd810f1ee
Author(s)
Olken, Benjamin A.
Date Issued
August 2015
Journal
Journal of Economic Perspectives
Publisher
American Economic Association (AEA)
Citation
Olken, Benjamin A. “ Promises and Perils of Pre-Analysis Plans.” Journal of Economic Perspectives 29, no. 3 (August 2015): 61-80. © 2016 American Economic Association.
Version
Final published version
Abstract
Imagine a nefarious researcher in economics who is only interested in finding a statistically significant result of an experiment. The researcher has 100 different variables he could examine, and the truth is that the experiment has no impact. By construction, the researcher should find an average of five of these variables statistically significantly different between the treatment group and the control group at the 5 percent level—after all, the exact definition of 5 percent significance implies that there will be a 5 percent false rejection rate of the null hypothesis that there is no difference between the groups. The nefarious researcher, who is interested only in showing that this experiment has an effect, chooses to report only the results on the five variables that pass the statistically significant threshold. If the researcher is interested in a particular sign of the result—that is, showing that this program “works” or “doesn’t work”— on average half of these results will go in the direction the researcher wants. Thus, if a researcher can discard or not report all the variables that do not agree with his desired outcome, the researcher is virtually guaranteed a few positive and statistically significant results, even if in fact the experiment has no effect.
MIT Department
Massachusetts Institute of Technology. Department of Economics
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1257/jep.29.3.61
