Abstract
The rapid expansion of the Large Language Model (LLM) ecosystem presents a significant challenge for developers: selecting the most cost-effective model for a specific task without sacrificing quality. Traditional routers often require expensive retraining to adapt to new models or data. We present Eagle, a novel, training-free routing framework that combines global ELO ratings with local, context-aware rankings. Eagle outperforms traditional machine-learning-based routers in accuracy while offering 100-200x faster updates and significantly lower initialization overhead.
Introduction
Developers currently face a "Goldilocks" dilemma: high-capability models like GPT-4o are prohibitively expensive for simple tasks, while smaller models are often too specialized for general use. LLM routers aim to solve this by acting as traffic controllers, directing queries to the most efficient model. However, existing solutions struggle with scalability, sparse feedback, and the need for real-time adaptation. Eagle addresses these hurdles by replacing traditional neural network training with a hybrid ELO-based ranking system that learns instantly from user feedback.
The Eagle Architecture
Eagle's architecture is designed for efficiency and rapid adaptation. Unlike traditional routers that require periodic retraining, Eagle utilizes a dual-module scoring system that updates in real-time:
- Eagle-Global: This module maintains a "general ability" score for every model in the pool using the ELO rating system. It converts sparse pairwise comparisons into a comprehensive global leaderboard.
- Eagle-Local: This module identifies specialized abilities by retrieving the $N$ most similar historical queries from a vector database. It then calculates a local ELO score based on model performance within that specific context.
The final routing decision is determined by a weighted sum of these scores: $$Score(X) = P \times Global(X) + (1 - P) \times Local(X)$$
The system then selects the highest-ranked model that satisfies the user's specified budget constraints. Because it only updates scalar scores and performs vector lookups, it requires zero traditional "training."
Experimental Results
Eagle was evaluated using RouterBench, a comprehensive benchmark covering tasks such as MMLU (general knowledge), MBPP (coding), and GSM8K (math).
Performance and Budget
A primary advantage of Eagle is its performance across varying budget levels. As shown in Figure 2, Eagle consistently provides higher performance than baseline routers (SVM, KNN, and MLP) across almost every "Willingness to Pay" threshold. This demonstrates its robustness in real-world scenarios where cost is a primary constraint.
Efficiency and Accuracy
- Superior Accuracy: Eagle achieved a 23.52% improvement in Area Under Curve (AUC) scores over SVM and roughly a 5% improvement over more complex MLP models.
- Blazing Fast Updates: In online learning scenarios where the router must adapt to new data, Eagle was 100-200x faster than traditional methods.
- Low Initialization Overhead: Eagle required only 4.8% of the time needed by baseline methods to set up initially.
Discussion and Limitations
Eagle demonstrates that competitive gaming metrics like ELO can be effectively repurposed for LLM evaluation. By blending global and local insights, Eagle recognizes that a model might be a mediocre generalist but a world-class specialist in specific domains like Python coding or medical advice.
However, the system's performance is sensitive to parameters like the number of neighbors ($N$) and the weighting factor ($P$). Additionally, as the historical query database grows to millions of entries, vector search latency may become a bottleneck. Future work will focus on optimizing these parameters and scaling the retrieval mechanism for high-volume production environments.
Summary updated