Improving Accuracy Predictions of Companion Classifiers for LLM Routing

Abstract

The increasing versatility of Large Language Models (LLMs) calls for developing effective routing systems to match tasks with the most suitable models, balancing accuracy and computational cost. This research introduces a novel meta-cascade routing framework that combines meta-routing, where a predictive model selects the appropriate LLM for a task, and cascading, where models are queried in sequence to optimize cost and performance. A critical component of this framework is the companion classifier, defined as a fine-tuned model trained to predict whether a particular LLM will generate an accurate response. We investigate whether incorporating features such as model responses into these classifiers can improve routing accuracy. Our preliminary experiments, using the Routerbench dataset, focus on training companion models that provide more stable and accurate routing decisions.