Routing Expert-Domain Questions by the Kind of Uncertainty

When a general-purpose language model faces a pharmacology exam question, an appellate law case, or a physics problem that needs a specific theorem, it can fail in several different ways. It may not know the governing principle. It may know the principle but botch the arithmetic. It may simply be thrown off by how the question is worded. Current systems that pick among models or add knowledge to the prompt mostly treat these failures as one thing, "the model is unsure," and respond with a single remedy: call a bigger model, retrieve more text, or convene a panel of models. This proposal tests a sharper hypothesis. The type of a model's uncertainty on a single question, measured on the fly, predicts which remedy will help. Missing knowledge should be sent to a domain specialist or a structured-knowledge pipeline. Computational shakiness should be sent to program-based solving. Wording sensitivity, or low uncertainty of any kind, should get no intervention at all.

The last case matters as much as the others. Several multi-model systems get worse on questions the base model already handles. In $n$-Musketeers, accuracy on a saturated logic task fell from 96.88% to 82.81% once expert models were added. Evidence on knowledge injection also conflicts. Injecting knowledge-graph triples helped a medical reader pass a pharmacist licensing exam (KMQA), while pasting theorem definitions into the prompt gave no significant gain on TheoremQA. Our claim is that both findings are averages over questions that needed different things. A per-question router that matches the remedy to the gap should beat every fixed policy, and it should beat a plain "escalate when unsure" rule at equal compute.

Core Claim and What Is New

We will build a router with three parts. First, for each incoming question it computes a small uncertainty profile of a generalist model: sampled answers under paraphrases of the question, answers under a clarified restatement, and answers when the computation is offloaded to code. Second, it maps that profile to one of four actions: answer directly, solve through a program, answer with injected structured knowledge, or hand off to a domain-specialist model. Third, it chooses the action that maximizes predicted accuracy gain minus a compute penalty,

$$ r^*(x) = \arg\max_{r \in \mathcal{R}} \; \hat{g}_r(u(x)) - \lambda\, c_r, $$

Here $x$ is the question and $u(x)$ is its uncertainty profile. $\mathcal{R}$ is the set of four remedies, with "answer directly" as the zero-cost null option. $\hat{g}_r$ is a small learned predictor of how much remedy $r$ improves correctness over answering directly. $c_r$ is the remedy's token cost, and $\lambda$ sets the accuracy–cost trade-off.

The uncertainty decomposition comes from Uncertainty Profiles for LLMs. That work splits uncertainty into surface-form, aleatoric, epistemic, and operational components using prompt perturbations. It then uses the resulting four-number fingerprint only at the level of whole tasks, matching a task's average profile to a model by cosine similarity, and it reports 5.5% NDCG gains in model selection. We move the same fingerprint to single questions, and we route to remedies of different kinds rather than to interchangeable models. That paper also reports leakage between categories, for example calculation noise inflating the epistemic estimate. This is the main technical risk, and we test it directly rather than assume it away.

The closest system at the intersection is EMRC. It classifies a medical question by department and difficulty, looks up an offline table of which language models scored best on that cell of a validation set, then runs confidence fusion, a judge model, and multi-round refinement. EMRC reaches 83.42% on MedQA. Its routing, however, depends on a coarse label plus the models' self-reported confidence, and the authors report degradation beyond four agents or two collaboration rounds. Our router differs in three ways. It conditions on how the generalist behaves on this question rather than on the question's category. It includes an explicit "do nothing" option. It chooses among qualitatively different interventions rather than among peer models.

Two other lines of work shape the design. They supply mechanisms, but they operate in settings we do not share. DynaSemble shows that simple statistics of each model's score distribution (mean and variance) can drive a small network that weights text-based against graph-based models per query. It gains up to 6.8 mean reciprocal rank points on WN18RR, but it is limited to completing knowledge-graph triples. We borrow its lesson that cheap distributional features suffice for per-query weighting, and we apply it to open question answering. FUSE weights 7 to 33 verifiers per query without labels using a moment-matching estimator. It is a post-hoc selector over already generated answers, so it cannot prevent a wasted generation. We use it as a strong competitor rather than a component.

Research Program

The work has three stages.

Stage 1: measure the per-question profile. For a fixed generalist we will compute, on each question, answer disagreement across $k$ paraphrases, disagreement under a clarified restatement, and the change in disagreement when the model writes and executes code instead of reasoning in text. The answer-type formats of TheoremQA and multiple-choice exams make agreement easy to measure. The model is Llama-3.3-70B, chosen so the FUSE comparison is on a matching generator.

Stage 2: build the remedies. The program remedy is program-of-thought prompting, with code written and executed. The knowledge remedy on the medical and pharmacy data injects retrieved medical knowledge-graph triples as pseudo-sentences, as in KMQA. On TheoremQA it supplies the theorem statement together with a worked application, since the bare statement alone was shown not to help. The specialist remedy uses domain-tuned models from the EMRC pool (code) for medicine and a law-tuned model for AppealCase.

Stage 3: fit and deploy the router. We fit $\hat{g}_r$ on a small labeled development split, sized like EMRC's validation use of MedQA. We also report a label-free variant that uses FUSE-style pseudo-labels in place of gold labels, because expert annotation is the bottleneck in these domains.

The analysis centers on an interaction test. Every remedy is run on every question, so each question gets an observed gain for each remedy. We then ask whether the high-epistemic, low-operational questions gain most from knowledge or specialist remedies, and whether the reverse questions gain most from program execution. If the four-way profile does not predict which remedy helps beyond predicting whether anything helps, the core claim fails. The router would then collapse into an ordinary uncertainty cascade, and we would report that outcome.

Baselines

The naive reference point is the generalist answering every question directly with chain-of-thought. Its strongest single-model counterpart is GPT-4 with program-of-thought, which TheoremQA reports at 51.0% accuracy. We will rerun that setting on our generator so the comparison is controlled, and we will quote the published 51.0% as an external anchor.

Fixed-remedy policies apply one remedy to every question: always program, always inject knowledge, always call the specialist. Each is run with the same prompts and retrieval as inside the router. Together these represent the static knowledge-injection position that KMQA defends. On the pharmacist exam, KMQA reports 61.8% accuracy against a 60% passing line, and we use it as the static-injection reference there. Beating these policies means exceeding the best single fixed remedy on each benchmark, not just their average.

The strongest existing routing system in medicine is EMRC, run from its released code with the same model pool and the same MedQA validation data for building its expertise table. Its reported results are 83.42% on MedQA, 71.91% on NEJMQA, and 74.45% on MMLU-Pro-Health. We require either matching its accuracy at a clearly lower token cost, or exceeding it at equal cost. We expect the cost advantage because EMRC runs up to four models over several rounds on every question.

The strongest label-free selector is FUSE, which reports 64.4% on GPQA Diamond with a Llama 3.3 70B generator, against 64.1% for a semi-supervised method given 5% labels. We will run FUSE on GPQA Diamond and TheoremQA with the verifier pool size tuned under the same total token budget our router consumes, so any gain is not bought with extra compute.

Two ablations isolate the key ingredient. The first gives the router a single scalar uncertainty (answer disagreement) instead of the decomposed profile, keeping everything else fixed. The second removes the "answer directly" option. The first tests remedy matching and the second tests the saturation fix.

Why not just…?

Why not just escalate whenever self-consistency is low? This is the most serious threat. A single disagreement threshold that sends unsure questions to the most expensive remedy is simple and may capture most of the gain. The scalar-uncertainty ablation is exactly this baseline, run at matched compute. If it comes within noise of the decomposed router on TheoremQA and MedQA, we will concede that uncertainty type adds nothing and report the cascade as the practical recommendation. The case for decomposition rests on the fact that remedies here are not ordered by strength. TheoremQA's finding that bare definitions do not help suggests a knowledge remedy wastes compute on a question that fails for arithmetic reasons. We expect the gap to show up most clearly in cost rather than peak accuracy.

Why not just use a bigger generalist? Scale alone has not closed these gaps in the reported evidence. GPT-4 reaches 51.0% on TheoremQA against 60–95% for undergraduates. On AppealCase, all 20 tested models, including GPT-4 and DeepSeek-V3, stay below 50% F1 on predicting reversal from first-instance documents. Improving low-resource QA argues that parameter count does not stand in for missing niche data. We still run a stronger generalist as the direct-answer option, so any benefit of scale is included rather than excluded.

Why not just always inject knowledge, as KMQA does? KMQA's own error analysis attributes 32% of its failures to retrieval noise. TheoremQA found that universal theorem injection gave no significant boost. The always-inject policy is a baseline here. The per-question gain matrix will show directly whether injection hurts a measurable fraction of questions the base model answers correctly, which is what would justify routing.

Why not just let several models debate, or ensemble them? EMRC reports degradation beyond four agents and two rounds, and $n$-Musketeers describes text debate as token-heavy. Static averaging is weak on the reported evidence: FUSE beats naive averaging and majority vote by 10 to 20 points on MMLU-Pro. We therefore compare against EMRC and FUSE rather than against plain voting.

Proposed Experiments and Decisive Checks

The benchmarks are TheoremQA across all four subject areas, MedQA, NEJMQA and MMLU-Pro-Health for medicine, NLPEC for pharmacy, GPQA Diamond for expert STEM, and first-instance reversal prediction on AppealCase for law. AppealCase is the case where no programmatic checker of reasoning exists, although the reversal labels allow scoring. The paper reports only that all models stay below 50% F1, so no exact per-model number is available. Metrics are accuracy (F1 for AppealCase), tokens consumed per question, and a harm rate: the fraction of questions the direct answer got right that the routed system gets wrong.

Four results settle the proposal. First, the uncertainty-type-by-remedy interaction in the per-question gain matrix must be significant on held-out questions. Second, the decomposed router must beat the scalar-uncertainty cascade at matched tokens on at least TheoremQA and MedQA. Third, on questions where the base model already answers correctly, the harm rate must fall well below that of the always-specialist policy. This directly tests the saturation degradation seen in $n$-Musketeers. Fourth, we check the stability of the decomposition by tracking how often the operational and epistemic scores move together on questions with known computational difficulty. If they move together heavily, the leakage reported by Uncertainty Profiles also rules out per-question use, and the proposal's mechanism is refuted rather than tuned around.