Clustering and visualizing solution variation in massive programming classes

Abstract

In large programming classes, a single problem may yield thousands of student solutions. Solutions can vary in correctness, approach, and readability. Understanding large-scale variation in solutions is a hard but important problem. For teachers, this variation could be a source of innovative new student solutions and instructive examples. Understanding solution variation could help teachers write better feedback, test cases, and evaluation rubrics. Theories of learning, e.g., analogical learning and variation theory, suggest that students would benefit from understanding the variation in the fellow student solutions as well. Even when there are many solutions to a problem, when a student is struggling in a large class, other students may have struggled along a similar solution path, hit the same bugs, and have hints based on that earned expertise. This thesis describes systems that exploit large-scale solution variation and have been evaluated using data or live deployments in on-campus or edX courses with thousands of students. OverCode visualizes thousands of programming solutions using static and dynamic analysis to cluster similar solutions. Compared to the status quo, OverCode lets teachers quickly develop a high-level view of student understanding and misconceptions and provide feedback that is relevant to more student solutions. Foobaz helps teachers give feedback at scale on a critical aspect of readability, i.e., variable naming. Foobaz displays the distribution of student-chosen names for each common variable in student solutions and, with a few teacher annotations, it generates personalized quizzes that help students learn from the good and bad naming choices of their peers. Finally, this thesis describes two complementary learnersourcing workflows that help students write hints for each other while reflecting on their own bugs and comparing their own solutions with other student solutions. These systems demonstrate how clustering and visualizing solution variation can help teachers directly respond to trends and outliers within student solutions, as well as help students help each other.