Eagle: Efficient Training-Free Router for Multi-LLM Inference

Zesen Zhao, Shuowei Jin, Z. Morley MaoarXiv 2024 · DOI · arXiv

Abstract

The rapid expansion of the Large Language Model (LLM) ecosystem presents a significant challenge for developers: selecting the most cost-effective model for a specific task without sacrificing quality. Traditional routers often require expensive retraining to adapt to new models or data. We present Eagle, a novel, training-free routing framework that combines global ELO ratings with local, context-aware rankings. Eagle outperforms traditional machine-learning-based routers in accuracy while offering 100-200x faster updates and significantly lower initialization overhead.

Introduction

Developers currently face a "Goldilocks" dilemma: high-capability models like GPT-4o are prohibitively expensive for simple tasks, while smaller models are often too specialized for general use. LLM routers aim to solve this by acting as traffic controllers, directing queries to the most efficient model. However, existing solutions struggle with scalability, sparse feedback, and the need for real-time adaptation. Eagle addresses these hurdles by replacing traditional neural network training with a hybrid ELO-based ranking system that learns instantly from user feedback.

The Eagle Architecture

Eagle's architecture is designed for efficiency and rapid adaptation. Unlike traditional routers that require periodic retraining, Eagle utilizes a dual-module scoring system that updates in real-time:

  1. Eagle-Global: This module maintains a "general ability" score for every model in the pool using the ELO rating system. It converts sparse pairwise comparisons into a comprehensive global leaderboard.
  2. Eagle-Local: This module identifies specialized abilities by retrieving the $N$ most similar historical queries from a vector database. It then calculates a local ELO score based on model performance within that specific context.
Eagle System Architecture

The final routing decision is determined by a weighted sum of these scores: $$Score(X) = P \times Global(X) + (1 - P) \times Local(X)$$

The system then selects the highest-ranked model that satisfies the user's specified budget constraints. Because it only updates scalar scores and performs vector lookups, it requires zero traditional "training."

Experimental Results

Eagle was evaluated using RouterBench, a comprehensive benchmark covering tasks such as MMLU (general knowledge), MBPP (coding), and GSM8K (math).

Performance and Budget

A primary advantage of Eagle is its performance across varying budget levels. As shown in Figure 2, Eagle consistently provides higher performance than baseline routers (SVM, KNN, and MLP) across almost every "Willingness to Pay" threshold. This demonstrates its robustness in real-world scenarios where cost is a primary constraint.

Performance vs. Willingness to Pay

Efficiency and Accuracy

  • Superior Accuracy: Eagle achieved a 23.52% improvement in Area Under Curve (AUC) scores over SVM and roughly a 5% improvement over more complex MLP models.
  • Blazing Fast Updates: In online learning scenarios where the router must adapt to new data, Eagle was 100-200x faster than traditional methods.
  • Low Initialization Overhead: Eagle required only 4.8% of the time needed by baseline methods to set up initially.

Discussion and Limitations

Eagle demonstrates that competitive gaming metrics like ELO can be effectively repurposed for LLM evaluation. By blending global and local insights, Eagle recognizes that a model might be a mediocre generalist but a world-class specialist in specific domains like Python coding or medical advice.

However, the system's performance is sensitive to parameters like the number of neighbors ($N$) and the weighting factor ($P$). Additionally, as the historical query database grows to millions of entries, vector search latency may become a bottleneck. Future work will focus on optimizing these parameters and scaling the retrieval mechanism for high-volume production environments.

Summary updated

Abstract

The proliferation of Large Language Models (LLMs) with varying capabilities and costs has created a need for efficient model selection in AI systems. LLM routers address this need by dynamically choosing the most suitable model for a given query based on task requirements and budget constraints. However, existing routers face challenges in scalability and real-time adaptation, particularly in high-volume online environments. We present Eagle, a novel LLM routing approach that combines global and local ELO ranking modules to overcome these limitations. By evaluating both general and specialized LLM abilities, Eagle provides a scalable, training-free solution that enhances model selection quality while reducing computational overhead. Our experiments across multiple datasets show Eagle consistently outperforms baseline methods, with improvements of up to 23.52 percent in Area Under Curve (AUC) scores. Moreover, Eagle demonstrates remarkable efficiency, requiring only 1/20 of baseline methods' time for initialization and 100 to 200 times faster incremental updates in online scenarios, making it well-suited for dynamic, high-volume online serving environments.
BibTeX
@misc{zhao2024eagle,
  author = {Zesen Zhao and Shuowei Jin and Z. Morley Mao},
  title = {{Eagle: Efficient Training-Free Router for Multi-LLM Inference}},
  year = {2024},
  eprint = {2409.15518},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15518},
}
Figure