Select-then-Route : Taxonomy guided Routing for LLMs

Soham Shah, Kumar ShridharEMNLP 2025

TL;DR

Large Language Models (LLMs) are powerful but no single model excels at everything. This paper introduces SELECT-THEN-ROUTE (STR), a two-stage framework that first narrows down the pool of relevant LLMs using a taxonomy and then uses a confidence-based cascade to select the best model. STR significantly improves accuracy and reduces costs compared to using individual LLMs or simpler routing methods.

What Problem Does This Paper Address?

While LLMs have achieved remarkable performance across various tasks, they often have specialized strengths. A naive approach of querying every available LLM for each task is prohibitively expensive in terms of both time and computational cost. Existing routing methods often fall short by either not adequately reducing the search space or by failing to account for the noise and cost-performance trade-offs inherent in LLM usage. The core challenge is to leverage the diversity of LLMs for better performance without incurring the full computational burden.

Methods

STR employs a two-stage approach:

  1. Taxonomy-guided Selection: This stage uses a lightweight classifier to map an incoming query to a semantic class (e.g., reasoning, code, summarization). Based on this class, it retrieves a small, high-recall pool of LLMs that are known to perform well for that specific category. This is akin to candidate generation in extreme classification. The paper proposes two methods for this classification: a slow but accurate LLM-based classifier and a faster embedding-based classifier, which are fused for better performance.
  2. Confidence-based Cascade: Within the selected pool of LLMs, a cascading strategy is used. It starts with the cheapest model and escalates to more expensive or capable models only when a multi-judge agreement test signals low confidence in the current model's output. This is similar to coarse-to-fine pruning. The confidence is assessed using a method called CASCADE, which combines multiple signals (logit-based confidence, reward model score, domain-specific verifier, and an LLM judge) to provide a robust confidence metric.

Datasets

The paper uses six public benchmarks across various domains for evaluation, including instruction following, formal reasoning, code generation, and constrained instruction following. They also mention constructing training and testing datasets that combine open-source datasets with proprietary enterprise data. The training set contains 15,023 samples, and the testing set has 1,567 samples.

Key Insights and Ideas

  • The power of structured reduction: The core idea is that reducing the decision space for LLM selection, first through taxonomy and then through confidence, is key to achieving both high accuracy and cost efficiency.
  • Taxonomy as a smart initial filter: Using a taxonomy to group queries and pre-select relevant LLMs is an effective way to avoid unnecessary computation on irrelevant models.
  • Cascading for cost-performance balance: The confidence-based cascade ensures that expensive models are only invoked when cheaper ones are insufficient, striking a balance between performance and cost.
  • Multi-signal judgment is crucial: The CASCADE method, which combines multiple confidence signals, provides a more reliable assessment of an LLM's output than a single metric.
  • Model diversity is key: The paper emphasizes that routing offers the biggest gains when the LLM pool contains models with diverse strengths.

Key Results

  • STR improved end-to-end accuracy from 91.7% (best single model) to 94.3% across six public benchmarks.
  • Inference cost was reduced by 4x compared to using the best single model.
  • STR outperformed the average LLM by 10.6% in terms of cost efficiency, costing $5.21 per 1000 prompt samples compared to $16.29.
  • The system offers a tunable cost-accuracy frontier, allowing users to adjust the trade-off based on their specific latency and budget constraints.

Limitations and Future Work

  • Reliance on manually defined taxonomy: The current taxonomy-based router depends on manually defined rules and categories, which requires expert knowledge and limits adaptability to new domains or unexpected query types.
  • Manual integration of new LLMs: Adding new LLMs to the STR model pool is currently a manual process.
  • Fixed judges: The judges selected for evaluation are fixed, meaning the judging strategy needs updating if evaluation models or task domains evolve.

The authors suggest future work on developing dynamic and adaptive judge selection mechanisms.

Summary updated

BibTeX
@inproceedings{shah2025select,
  author = {Soham Shah and Kumar Shridhar},
  title = {{Select-then-Route : Taxonomy guided Routing for LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure