2026-09-18T20:27:26Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1627362025-09-19T04:49:50Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Liang, Paul Pu
Lee, Jimin
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2025-09-18T14:29:49Z
2025-09-18T14:29:49Z
2025-05
2025-06-23T14:02:43.209Z
https://hdl.handle.net/1721.1/162736
Effective reasoning often requires more than text or language. It requires visualizing, drawing, gesturing, and interacting for both humans and artificial intelligence (AI). Specifically in educational subjects, such as geometry and graphs, visual tools like auxiliary annotations and drawings can greatly help students understand abstract theories. This thesis explores and suggests how multimodal interaction between humans and AI helps humans engage with the system more naturally and effectively, leading to improved problem-solving in mathematical settings. Recent large multimodal models (LMMs) have the ability to facilitate collaborative reasoning by supporting textual, visual, and interactive inputs, diversifying methods of communication between humans and AI. Utilizing such advancements, this thesis also dives into the development of Interactive Sketchpad, a tutoring system that combines language-based explanations with interactive visualizations to enhance learning. It also reviews findings from user studies with Interactive Sketchpad, demonstrating that multimodality contributes to user task comprehension and engagement levels. Together, these contributions can reframe the role of AI in education as a visual and interactive collaborator that supports deeper reasoning rather than simply providing answers. Furthermore, this work demonstrates the potential of multimodal human-AI systems in fostering engagement and scaling personalized, visual learning across domains.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
A Pedagogical Multimodal System for Mathematical Problem-Solving and Visual Reasoning
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="be532a97-01b1-4b35-a2e2-ae1b269d0fce">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>A Pedagogical Multimodal System for Mathematical Problem-Solving and Visual Reasoning</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2025-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Lee, Jimin</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Effective reasoning often requires more than text or language. It requires visualizing, drawing, gesturing, and interacting for both humans and artificial intelligence (AI). Specifically in educational subjects, such as geometry and graphs, visual tools like auxiliary annotations and drawings can greatly help students understand abstract theories. This thesis explores and suggests how multimodal interaction between humans and AI helps humans engage with the system more naturally and effectively, leading to improved problem-solving in mathematical settings. Recent large multimodal models (LMMs) have the ability to facilitate collaborative reasoning by supporting textual, visual, and interactive inputs, diversifying methods of communication between humans and AI. Utilizing such advancements, this thesis also dives into the development of Interactive Sketchpad, a tutoring system that combines language-based explanations with interactive visualizations to enhance learning. It also reviews findings from user studies with Interactive Sketchpad, demonstrating that multimodality contributes to user task comprehension and engagement levels. Together, these contributions can reframe the role of AI in education as a visual and interactive collaborator that supports deeper reasoning rather than simply providing answers. Furthermore, this work demonstrates the potential of multimodal human-AI systems in fostering engagement and scaling personalized, visual learning across domains.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>