Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routing

Kv Aditya Srivatsa, Kaushal Kumar Maurya, Ekaterina KochmarNAACL 2024

TL;DR

This paper explores "LLM routing," a method to efficiently select the single best Large Language Model (LLM) for a given query from a pool of diverse LLMs. While routing shows promise for improving performance and maintaining low latency, the study reveals that current approaches struggle to consistently outperform top individual LLMs, highlighting the need for more robust routing models and larger datasets.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has led to a proliferation of powerful, yet diverse, models. No single LLM excels across all tasks and benchmarks; instead, different LLMs exhibit varying domain expertise. This raises a crucial question: can we efficiently leverage this diversity by directing each input query to the most suitable LLM, rather than relying on a single model or computationally expensive ensembling methods? This paper tackles this challenge by investigating "LLM routing," aiming to achieve better performance and efficiency by intelligently selecting a single LLM for each query, while keeping inference latency comparable to that of individual LLMs.

Methods

The authors propose and evaluate two primary approaches for LLM routing:

  • Classification-Based Routing: This method treats LLM routing as a classification task. A classifier is trained to predict which LLMs are capable of solving a given query. Two types of classifiers were explored:
  • Multi-label Classifier (MLC): Predicts all suitable LLMs in a single step.
  • Separate Classifiers (SC): Uses individual binary classifiers for each LLM. The best-performing classifier was a fine-tuned RoBERTa model. Based on the classifier's confidence scores, several routing policies were devised: ArgMax (highest confidence), Random (random selection from high-confidence LLMs), Prediction (regressing to a target confidence score), and Sorted Prediction (ranking LLMs by performance).

  • Clustering-Based Routing: This approach leverages similarities between queries. A KMeans clustering model is trained on query-specific features (TF-IDF or RoBERTa embeddings) to group similar queries. For each cluster, the best-performing LLM during training is identified, and queries in that cluster are routed to that LLM during inference.

Datasets

The study utilizes two challenging reasoning benchmarks:

  • GSM8K: A dataset of 8,792 grade-school level math word problems.
  • MMLU (Massive Multitask Language Understanding): A dataset comprising 15,000 multiple-choice questions across 57 subjects, covering STEM, humanities, and social sciences.

The authors remapped the MMLU data splits to align with GSM8K's distribution for consistency.

Key Insights and Ideas

  • Routing is Promising, but Not a Silver Bullet: While LLM routing demonstrates potential for improved performance over weaker LLMs and maintains comparable latency to individual models, it doesn't consistently surpass the performance of the top-performing LLMs in the pool.
  • Data Scarcity is a Bottleneck: The limited size of the training data for the routing classifiers is identified as a significant factor hindering the achievement of higher accuracy scores, potentially leading to classifier overfitting.
  • Diverse LLMs Offer Complementary Strengths: The study confirms that combining LLMs, even those with lower individual performance, can lead to better overall results than individual models, underscoring the value of model diversity.
  • Latency is Maintained: A key advantage of the proposed routing approach is its ability to keep inference latency at or below that of the best individual LLM, a critical factor for practical deployment.
  • Clustering Assumption May Not Hold: The assumption that LLM performance remains consistent across similar query clusters in both training and testing phases did not hold for a significant portion of the clusters, limiting the effectiveness of the clustering-based routing.

Key Results

  • The Oracle model (theoretical best performance) achieved accuracies of 87.18% on GSM8K and 89.15% on MMLU.
  • Individual top-performing LLMs like gemma-7b reached accuracies of 71.11% (GSM8K) and 63.85% (MMLU).
  • The proposed MLC+ArgMax policy achieved accuracies of 67.62% (GSM8K) and 62.28% (MMLU), which were better than many individual LLMs but generally lower than the top performers.
  • The clustering-based routing methods (TF-IDF and RoBERTa) achieved accuracies around 61.76% on both datasets, performing similarly to the MLC+ArgMax model.
  • In terms of latency, the routing models generally maintained inference times comparable to or lower than the best individual LLMs. For instance, MLC+ArgMax had a latency of 4.76 seconds on GSM8K, while gemma-7b had 7.10 seconds.

Limitations and Future Work

Limitations: * Limited Training Data: The small datasets used for training routing classifiers led to suboptimal performance and potential overfitting. * Answer Extraction Imperfections: While robust, the answer extraction algorithm could only successfully extract viable answers for 83-95% of queries, with some extracted answers being incorrect. * Effectiveness with Highly Diverse LLMs: The proposed model is more effective with equally capable LLMs; performance degrades when there are significant capability differences. * Memory Overhead: Frequent switching between LLMs can necessitate loading multiple models into memory, posing a memory challenge.

Future Work: * Larger Datasets: Collecting more extensive datasets for LLM routing is crucial for improving classifier performance. * Novel Routing Models: Developing more sophisticated models to accommodate LLMs with diverse capabilities. * Improved Routing Policies: Designing better policies that leverage confidence scores more effectively. * LLM-Specific Features: Incorporating LLM-specific features to enhance routing model accuracy. * Scaling Up: Expanding the approach to include a wider range of LLMs and benchmarks.

Summary updated

BibTeX
@inproceedings{srivatsa2024harnessing,
  author = {Kv Aditya Srivatsa and Kaushal Kumar Maurya and Ekaterina Kochmar},
  title = {{Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure