Using learning analytics to evaluate design changes in MOOCs : a case study on assessing course pacing

Abstract

Experimentation on course design in MOOCs can determine causal factors that promote learning and can identify aspects of the course where revision is needed. The presence of heterogeneous samples of learners, the difficulty of defining success metrics, and the lack of shared cross-course data are few of the challenges course designers face to evaluate MOOCs. In this thesis, we present a data-driven framework to evaluate design changes in MOOCs. We explore a change from multiple angles -process, proficiency, and perception- and apply various analytical methods -temporal, causal and predictiveto map out the outcome of instruction along multiple dimensions of learning. We demonstrate the application of this framework by evaluating course pacing on a repeated run of a supply chain MOOC by MITx. Self-pacing caused completion rate (-6%), pass rate (-10%), and engagement score (-7%) to drop, although students' satisfaction with course remained unchanged. The impact of pacing on students' outcome was not uniform with some experiencing no change while others encountering a steep fall. The most striking difference was seen in the longitudinal trajectories, with instructor-paced students mostly taking the same trajectory and self-paced students pursuing their own individually paced trajectories. We showed that these trajectories are correlated with student grade, and students with certain characteristics are inclined to pursue a specific trajectory. From these and other observations, we were able to provide practical guidance to course designers on what instructional materials and practices are satisfactory and where change is needed.