Recognizing Speech with Large Language Models
Name
zeitoun-zeitoun-sm-eecs-2023-thesis.pdf
Description
Thesis PDF
Size
1.03 MB
Format
Adobe PDF
Checksum (MD5)
4edbb99b453b769f2e9f5ba1686c572b
Author(s)
Zeitoun, Abbas
Advisor(s)
Kim, Yoon
Date Issued
June 2023
Publisher
Massachusetts Institute of Technology
Abstract
Recent work has shown that large language models can be made to parse the contents of non-text embeddings and use those contents to perform various tasks. However, work focusing on audio inputs to large language models has thus far focused on either training a joint audio-text model from scratch on a lot of data or on training the model to perform surface-level audio-text classification tasks. In this work, we show that a pretrained T5 encoder-decoder language model fine-tuned on as little as 10 hours of speech data can transcribe the contents of input audio embeddings and even outperforms a specialized baseline speech-to-text model at transcribing more difficult speech utterances. The resulting model serves as a first step towards language models that can manipulate audio inputs just as well as text inputs and can leverage the additional information in audio inputs to perform tasks that are not possible with text inputs alone.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
