Large Language Model Routing with Benchmark Datasets
Name
ou-aou-meng-eecs-2024-thesis.pdf
Description
Thesis PDF
Size
2.33 MB
Format
Adobe PDF
Checksum (MD5)
050f443100c99cda80d5c17a82631726
Author(s)
Ou, Anthony C.
Advisor(s)
Thompson, Neil
Date Issued
February 2024
Publisher
Massachusetts Institute of Technology
Abstract
There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. With a new dataset, it can be difficult to determine which LLM is best suited to the task. In this work we will address the challenges associated with selecting the best LLM model out of a collection for a new task. To do so, benchmark datasets are repurposed to learn a “router” model for this LLM selection, such that the “router” model will solve a collection of binary classification tasks. This work will demonstrate the utility and limitations of learning model routers from various benchmark datasets, where performance is improved upon using any single model for all tasks.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
