Improving Introductory Computer Science Students’ Programming Process When Using a Generative AI Tutor (PyTutor)

Abstract

This thesis examined students’ programming process while using PyTutor, a generative AI tutor for introductory computer science students. This thesis had the research questions: (1) How does the process of test case creation, with or without PyTutor’s Test Case Runner, impact students’ programming process while using PyTutor? (2) How can prompt engineering of PyTutor’s system prompt be leveraged to improve AI Chat response quality with respect to: (a) reducing the amount of code revealed in the answer, (b) improving the conciseness of responses, and (c) having the AI chat give the student test cases as a tool to understand code correctness? (3) How does PyTutor’s responses from the updated prompt affect the programming process for computer science students? A key finding from a focus group in the first stage (n=9) was apart from test cases and was that the majority of participants who asked questions to PyTutor received at least three lines of code, unideal for PyTutor’s pedagogical purpose. This discovery inspired the next phase of this thesis of prompt engineering PyTutor, which resulted in an updated prompt. Responses from the both the updated prompt and the original prompt were scored using an evaluation rubric. For the Students thinking through problem category of the evaluation rubric, it was statistically significant that the distribution of points for responses from the updated prompt was greater than the distribution of points for responses from the original prompt. Finally, participants were asked to solve a programming problem using either PyTutor with the updated prompt (n=10) or PyTutor with the original prompt (n=2). Across the focus groups from the first and final stage, I found that fewer participants who used PyTutor with the updated prompt received at least three lines of code. Furthermore, participants who used PyTutor with the updated prompt required a greater number of messages to first receive three lines of code. Additionally, all four participants who received at least three lines of code from PyTutor with the updated prompt asked majority high-level questions. As participant feedback suggested that PyTutor’s responses for high-level questions could be repetitive, this data highlights a new direction of improving PyTutor’s responses when answering high-level questions to benefit students’ programming process.