Validating the pre/post-test in a MOOC environment

Abstract

A standard method for measuring learning is to administer the same assessment before and after instruction. This pre/post-test technique is widely used in education research and has been used in our introductory physics MOOC to measure learning. One potential weakness of this paradigm is that post-test performance gains may result from exposure on the pre-test instead of instruction. This possibility is exacerbated in MOOCs where students receive multiple attempts per item, instant correct/incorrect feedback, and unlimited time (until the due date). To find the size of this problem in our recent MOOCs, we split the student population into two groups, each of which received identical post-tests but different subsets of post-test items on their group pre-test. We report a small overall advantage (2.9% ± 1.7%) on post-test items due to pre-test exposure. However, this advantage is not robust and is strongly diminished when one obviously anomalous item is removed.