2026-09-19T20:59:51Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1516082023-08-01T03:19:54Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Oliva, Aude
Feris, Rogerio
Purohit, Sonia
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2023-07-31T19:52:20Z
2023-07-31T19:52:20Z
2023-06
2023-06-06T16:35:17.730Z
https://hdl.handle.net/1721.1/151608
Automated visual understanding is an essential part of the sports industry, particularly in the context of major sports tournaments. The scale of generated video footage necessitates the use of automated systems to generate insights and enhance fan experiences. One area where this is particularly challenging is commentary, which requires detailed information about play-by-play action, a task that cannot be efficiently carried out by human commentators at scale.
We tackle this problem for grand-slam tennis through an IBM partnership with the Championships, Wimbledon. This thesis introduces a novel system that utilizes computer vision to extract play-by-play metadata and convert it into fluent commentary using large language models. Our computer vision module utilizes a single camera feed to understand every detail of the game – court and net detection, player and ball tracking, player poses, and fine-grained shot classification, all in near-real-time. This metadata is then combined with additional information from other modalities, such as crowd audio and radar-measured ball speed, and fed into a "data2text" large language model to generate commentary in natural language.
Our system not only supports the narration of match content at scale, but also powers the collection of additional metadata to facilitate additional match insights in the future.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
AI Commentator: Narrating Sports Games through
Multimodal Perception and Large Language Models
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="be75a659-2cbb-4c17-bd8e-e72bb99f91c6">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>AI Commentator: Narrating Sports Games through
Multimodal Perception and Large Language Models</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2023-06</PublicationDate>
<Authors>
<Author>
<DisplayName>Purohit, Sonia</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Automated visual understanding is an essential part of the sports industry, particularly in the context of major sports tournaments. The scale of generated video footage necessitates the use of automated systems to generate insights and enhance fan experiences. One area where this is particularly challenging is commentary, which requires detailed information about play-by-play action, a task that cannot be efficiently carried out by human commentators at scale.
We tackle this problem for grand-slam tennis through an IBM partnership with the Championships, Wimbledon. This thesis introduces a novel system that utilizes computer vision to extract play-by-play metadata and convert it into fluent commentary using large language models. Our computer vision module utilizes a single camera feed to understand every detail of the game – court and net detection, player and ball tracking, player poses, and fine-grained shot classification, all in near-real-time. This metadata is then combined with additional information from other modalities, such as crowd audio and radar-measured ball speed, and fed into a "data2text" large language model to generate commentary in natural language.
Our system not only supports the narration of match content at scale, but also powers the collection of additional metadata to facilitate additional match insights in the future.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>