Examining conversational programming design needs with Convo, a voice-first conversational programming system using natural language

Abstract

The emergence of voice-based technology and voice assistants has paved the way for new opportunities to democratize learning computational thinking and programming skills, especially for people who do not have access to the traditional programming experience, due to external circumstances or disabilities. In this thesis, I created Convo, a voice-first conversational programming system that allows users to create programs and develop programming skills simply through natural interactions and conversations with a programming agent and voice feedback. Additionally, I conducted a user study to study the effectiveness of voice-first conversational programming systems like Convo as well as receive user feedback to explore the design needs of such systems. The user study involved forty-five participants ranging from fourteen to sixty-five years old completing tasks in Convo using and comparing three different input modalities - using just voice inputs; using just text inputs; and using both voice and text inputs. The participants answered questions about their experiences with each input modality and general feedback on conversational programming systems. Results showed that participants have preference towards text and found voice-based programming the most difficult to use among the three input modalities. However many participants, especially those new to programming, saw the value and future potential of voice-based programming, especially when speech recognition becomes more accurate. Additionally, I created design recommendations based on the results for conversational programming systems, including being flexible and accessible, reducing cognitive load and improving speech recognition and natural language understanding.