Checking a scientific claim against a large literature is usually done by splitting the question into smaller sub-questions. For "does drug X reduce mortality in adult humans," the sub-questions might be: which population, which intervention, which outcome, and does each retrieved paper match on all three. Each sub-question is then answered by some mix of retrieval and a language model. A cheap model answers most sub-questions correctly. When it gets one wrong, a verifier catches the error, and the system pays to redo the step with an expensive model. Today that redo is treated as a local event. In a decomposition tree it is not local: a wrong answer at an upstream node, such as the extracted population, makes every downstream sub-question built on it worthless, so they must all be re-run. The true cost of a cheap-model failure therefore depends on where the node sits in the tree.
Logistics planners have handled this structure for decades with two-stage stochastic programming. A first-stage decision is committed before uncertainty is revealed, and a second-stage "recourse" corrects it afterward at a penalty. The plan minimizes upfront cost plus expected recourse cost. We propose treating the assignment of model tiers and retrieval depth to the nodes of a query tree as exactly such a first-stage decision. The recourse is verifier-triggered escalation, whose penalty includes re-running the node's descendants. The concrete prediction is that a planner aware of these recourse costs will put expensive models on upstream, high-fan-out, failure-prone nodes and cheap models on leaves. At matched cost, it will beat both whole-query routing and per-node confidence cascades on open-domain scientific claim verification.
Core Claim
The gap is that existing cost-saving methods for language models either route whole queries or fall back one call at a time, and none prices the downstream damage of a failure inside a decomposition.
Three existing methods each supply a piece but miss this pricing:
- Hybrid LLM routes each entire query between a small and a large model. It uses a router that predicts the quality gap between them. It does not decompose queries and has no fallback after generation.
- OCaTS adds the fallback. A cheap student answers when distance and entropy filters pass, and the teacher model is called otherwise. It works on flat classification, where one failure costs one teacher call.
- Query Decomposition for RAG: Balancing Exploration-Exploitation budgets retrieval over sub-query trees using bandits. Child nodes inherit beliefs from their parents. It counts only document relevance, and it treats downstream generation cost as uniform with no recourse.
The planner we propose differs from all three in one place: the price of failure at node $i$ includes the cost of re-executing its descendant set.
The method side comes from Planning hinterland container transportation in congested deep-sea terminals. That paper formulates barge planning as a two-stage problem. It then derives a closed-form per-unit limit from optimality conditions on a single-barge subproblem. With that limit, a small integer program reaches within 1% of full sample average approximation (SAA), the approach of solving the stochastic program exactly over many sampled scenarios. Solve time drops from hours to seconds. We transfer both the formulation and the trick of deriving a per-node rule.
Concretely, the plan is a vector $x$ that assigns each node $i$ a model tier and a retrieval depth. The random outcome $\xi$ records which nodes the verifier flags. The plan solves
$$\min_{x} \; c^\top x + \mathbb{E}_{\xi}\big[Q(x,\xi)\big] \quad \text{s.t.} \quad c^\top x + \mathbb{E}_{\xi}[Q(x,\xi)] \le B,$$
Here $c^\top x$ is the upfront token cost, $Q(x,\xi)$ is the cost of escalations and descendant re-runs under outcome $\xi$, and $B$ is the query's compute budget.
Relaxing to a single node gives a simple rule. Let:
- $p_i$ be the cheap model's failure probability at node $i$;
- $r$ be the verifier's recall on failures;
- $c_s$ and $c_L$ be the per-call costs of the small and large models;
- $C(D_i)$ be the cost of re-running node $i$'s descendants;
- $\lambda$ be the penalty for an undetected wrong answer.
The rule is to commit the large model upfront when
$$c_L - c_s < p_i\big[\, r\,(c_L + C(D_i)) + (1-r)\,\lambda \,\big].$$
When a node has no descendants, $C(D_i)=0$, and the rule reduces to an ordinary cascade threshold. That gives a built-in control: on flat decompositions the proposal should show no advantage. We take the large model's own failure rate as negligible here and model it explicitly in the full program.
The verifier term matters because triggers are noisy. Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis shows that different verifiers judging the same review output report unsupported-citation rates from 3% to 18%, with negative-specific agreement of only 0.27 to 0.30. That paper constrains our design: we cannot take $r$ as given. We will therefore use that paper's split-conformal guard, which calibrates a threshold so that unsupported claims pass with probability at most $\alpha$, as the recourse trigger. We then estimate $r$ on held-out data.
Research Program
Decomposition. We build a claim-verification pipeline over the 500,000-abstract corpus of SciFact-Open. Each claim is decomposed into a fixed-schema tree. The root node extracts the claim's population, intervention, and outcome. Mid-level nodes retrieve and screen candidate abstracts for each element. Leaf nodes make support or refute judgments per abstract, and an aggregation node combines them.
This schema is motivated directly by SciFact-Open's diagnosis. In 44% of its annotated evidence pairs the evidence is more or less specific than the claim, which is exactly what a population-extraction node addresses. Roughly 20% of claims have conflicting published evidence, which is the aggregation node's job. We fix the decomposition rather than learning it, so that the only variable is how compute is allocated across the tree.
Failure model. A small encoder predicts $p_i$ from the node's input, trained the way the Hybrid LLM router is trained: compare small-model and large-model outputs, sampled several times on training data. To build the scenarios that full SAA needs, we sample joint failure patterns from these per-node predictions plus a learned correlation. The correlation term captures, for example, that nodes sharing a weak retrieval pool tend to fail together.
Planners compared. We solve the problem three ways:
- full SAA over sampled scenarios, run offline and used as the quality reference;
- the closed-form per-node rule combined with a small integer program that enforces the budget;
- an "expectational" plan that replaces each $p_i$ with its mean and ignores recourse.
The gap between the expectational and stochastic plans is our analogue of the hinterland paper's value of stochastic planning, which there improved the objective by 6% to 9%.
Secondary test bed. We repeat the setup on QASPER, a LongBench task of long-context question answering over full scientific papers, to check transfer beyond abstracts.
Baselines
Naive reference points. We use two: "always large model" at every node, which sets the quality ceiling and the cost ceiling, and "always small model" with no recourse, which sets the cost floor. As a non-cost-aware accuracy reference we include MultiVerS, which SciFact-Open reports at 52.4 F1 on SciFact-Open, down from 72.5 F1 on closed SciFact. Our pipeline differs in architecture, so this number anchors plausibility rather than serving as the comparison that decides the claim.
Strongest competitor: per-node confidence cascade. Every node first runs the small model. The node escalates to the large model whenever the same conformal-guarded verifier flags it, which is the OCaTS fallback logic applied node by node. This is the real threat: it already exploits the fact that most sub-questions are easy, and it uses identical models, verifier, and retrieval. The cascade's threshold gets the same tuning budget as our planner's two cost parameters, tuned on the same development split.
Second competitor: per-node quality-gap routing. The Hybrid LLM router is applied to each node independently, with no descendant term. Hybrid LLM reports a 22% reduction in large-model calls at a 1% quality drop, measured by BART score on MixInstruct. We found no reported result on Lacuna for any per-node or decomposition-aware routing on SciFact-Open or QASPER, so we will establish these numbers ourselves.
Retrieval-budget baseline. We run hierarchical bandit retrieval from Query Decomposition for RAG with a fixed model tier at every node. This isolates whether the gains come from model allocation or from retrieval allocation.
Key ablations. The first ablation sets $C(D_i)=0$ in our rule, which isolates the descendant-cost term and should collapse our planner to the cascade. The second replaces the rule with a pure depth heuristic, "large model at the top two levels." If the depth heuristic matches our planner, the planner adds nothing beyond depth.
What counts as winning. Beating the cascade means a higher F1 at matched large-model token spend, or at least 15% less large-model spend at matched F1, holding on both SciFact-Open and QASPER. Both comparisons are averaged over five seeds with bootstrap intervals.
Why not just…?
Why not just run a confidence cascade at every node? A cascade is the optimal policy when failures are independent and each redo costs one call. Neither condition holds in a tree, because an upstream failure caught late wastes every descendant already computed. The descendant-cost ablation tests this directly. If the cascade matches our planner on deep trees, then descendant cost does not matter in practice and the central claim is wrong. We treat that as a live possibility. The flat-decomposition control, where both methods should tie, keeps the comparison honest.
Why not just route the whole claim once, as Hybrid LLM does? Whole-query routing must send an entire claim to the expensive model whenever any part of it is hard. Hybrid LLM's own analysis notes this limitation: a query with one hard sub-task goes wholly to the large model. Our baselines include whole-query routing, so the size of the gain from mixing tiers within a query will be measured, not assumed.
Why not just use the big model only near the root? A depth heuristic captures part of the intuition for free and is a serious threat. The rule improves on it only if failure probability and fan-out vary enough across claims that a fixed depth cutoff misallocates compute. Examples are a trivially extractable population paired with a heavily contested outcome. We will report the per-claim variance of $p_i$ and $C(D_i)$ to explain whatever gap appears. If the heuristic ties our planner, we will recommend the heuristic.
Why not just improve the small model so recourse is rare? Fine-tuning shrinks $p_i$ but does not remove the open-domain shift. SciFact-Open shows at least a 15-point F1 drop for strong verifiers moving from curated to open corpora. Any residual failure still carries descendant cost. Planning composes with a better small model rather than competing with it, and we will rerun the planner with a fine-tuned small model to confirm this.
Proposed Experiments
Main experiment. We trace F1 against large-model token spend on SciFact-Open's 279 claims and on QASPER, for every planner and baseline above. We also report wall-clock latency, because sequential escalation adds delay that upfront commitment avoids.
Solver check. We report the closed-form planner's optimality gap and solve time against full SAA on trees small enough to solve exactly. The hinterland result, under 1% gap with seconds rather than hours, is the bar.
Value of stochastic planning. We report the cost difference between the stochastic and expectational plans at matched F1.
Decisive checks. Three results decide the claim:
- The advantage over the cascade should grow with tree depth and fan-out, and vanish on flat decompositions.
- The advantage should shrink as verifier recall $r$ falls. We vary $r$ by changing the conformal $\alpha$, since the rule predicts exactly this interaction.
- Plans calibrated on SciFact-Open should keep their ordering against baselines on QASPER.
If the first check fails, the descendant-cost mechanism is not doing the work, and the proposal reduces to a well-tuned cascade.
References
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models
- Query Decomposition for RAG: Balancing Exploration-Exploitation
- Planning hinterland container transportation in congested deep-sea terminals
- Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
- SciFact-Open: Towards open-domain scientific claim verification