When a cheap language model answers a question about a long document, its answer usually echoes the document. You can find exact runs of the answer's words inside the prompt, and the positions of those runs show which parts of the document the model actually used. We propose computing this "evidence footprint" with a fast exact-match index over the prompt. The footprint records how much of the answer is grounded in the prompt and how spread out the grounding is. We then use it to decide what happens next. The cheap answer can be accepted. The query can be escalated to a large model with the full document. Or the query can be escalated to the large model with only retrieved passages. The footprint costs milliseconds, needs only text, and works even when the large model sits behind a closed API that exposes no internal probabilities.
The third option is what makes this more than a better confidence score. Model cascades send a query to a cheap model first and escalate to an expensive one only when needed. Nearly all of them assume escalation never hurts: whenever the small model is right, the large model is right too. Long-context evaluations contradict this. Frontier models lose accuracy as inputs grow, and past roughly 128k tokens, feeding a model retrieved chunks beats feeding it the whole document. A router for long inputs therefore has two jobs. It must detect when the small model failed, and it must also predict whether the large model with full context would fail in turn. We argue that the footprint, which measures how dispersed the dependencies are, carries information for both decisions that neither query length nor the small model's own confidence provides.
Core Claim
On long-context question answering, a footprint signal computed by exact matching between the small model's output and its prompt predicts two things better than small-model confidence or input length alone. It predicts the small model's errors. It also predicts the cases where the large model with full context is no better or worse, which we call dominance violations. A three-way routing policy built on this signal, with thresholds set by conformal calibration, should reach a better accuracy-per-cost trade-off than two-way cascades on LongBench, LaRA, and 100-LongBench.
The concrete signal works as follows. For each output token $t$, let $m_t$ be the length of the longest run of output tokens ending at $t$ that occurs verbatim in the prompt, and let $p_t$ be the prompt position of that match. Fix a minimum match length $k$ and keep the set $G$ of tokens with $m_t \ge k$. From $G$ we read off three quantities. Coverage $C = |G| / T$ is the fraction of the $T$ output tokens that are grounded. The number of clusters $K$ counts distinct groups of matched positions after merging those within a window $w$. Spread is
$$D = \frac{\max_{t \in G} p_t - \min_{t \in G} p_t}{L},$$
where $L$ is the prompt length. These features, along with $L$ and optionally the small model's confidence, feed a small classifier with two output heads: "small model wrong" and "large model with full context beats large model with retrieved context." The intuition is that low coverage signals an answer synthesized from parametric memory, which risks hallucination. High coverage concentrated in one cluster, on a question whose template implies multiple hops, signals that the small model latched onto one local passage. High spread at very long $L$ marks exactly the dispersed-dependency regime where full-context processing degrades.
Prior Work and What It Leaves Open
Cascade research supplies the routing machinery but treats difficulty as a property of the query text or the model's confidence. RouteLLM trains routers on preference battles. Gatekeeper fine-tunes the small model so that its wrong answers become uncertain. C3PO calibrates exit thresholds with conformal prediction under a cost budget. STEER routes individual reasoning steps using logit confidence. All four assume the large model dominates, and none uses any feature describing where in a long prompt the evidence lives. We reuse C3PO's conformal calibration as our method for setting thresholds, but we change both the signal and the action space.
Work on n-gram matching supplies the measurement tool. Infini-gram shows that suffix-array lookups are millisecond-fast and that long exact matches are highly predictive, with over 75% next-token accuracy when the match length is at least 16. N-gram trie speculative decoding and ANPD build n-gram structures over the prompt in about 10 ms to draft tokens for a single model. All three use matches either to predict tokens or to accelerate a fixed model. None uses the match pattern as a judgment about whether an answer should be trusted, which is our use.
Long-context evaluations motivate the three-way action space. LongBench reports a 17% accuracy drop for GPT-3.5-Turbo-16k from the 0–4k range to the 8k+ range. LaRA finds that at 128k tokens retrieval beats full context by 3.68% on average and by 38.12% for Llama-3.2-3B. LaRA also suggests hybrid routing between retrieval and full context but does not build a cascade. 100-LongBench shows that standard benchmarks conflate base knowledge with long-context ability and provides length-controlled variants. Those variants are what let us separate the footprint from mere length. Finally, L2M shows that the information shared between large text blocks grows sub-linearly with length. That result motivates treating dependency dispersion, rather than raw token count, as the quantity that matters.
Research Program
We first build the measurement. For each prompt we construct a suffix array over its tokens. We run the small model to get an answer and compute $m_t$, $p_t$, $C$, $K$, and $D$. We choose $k$ and $w$ on a held-out split. For every instance we also run three reference systems: the large model with full context, the large model with top-$k$ retrieved chunks, and the small model with retrieved chunks. This gives the full outcome table needed to train and score any routing policy offline.
Next we run a diagnostic study, before any policy is built, that asks whether the footprint carries information at all. On the length-controlled 100-LongBench variants we hold $L$ fixed and test whether $D$ and $K$ predict small-model and large-model errors. If they do not, the idea fails early and cheaply.
Then we build the policy. We train a gradient-boosted classifier, deliberately small so that its own cost is negligible, with the two heads described above. We map its outputs to three actions: accept the small answer, escalate with full context, or escalate with retrieved context. Following C3PO, we set the thresholds by conformal calibration so that the escalation rate stays under a stated budget with a stated risk level.
Baselines
The naive reference points are "always small model with full context," "always large model with full context," and "always large model with retrieved chunks." These bound the accuracy-cost plane, and the third already captures LaRA's long-input advantage for free.
The strongest existing competitor is a confidence-threshold cascade in the style of Gatekeeper and C3PO. The small model's sequence-level confidence, or self-consistency agreement over five samples as in C3PO and Mixture of Thought cascades, decides escalation to the large model with full context, and the threshold is set by the same conformal procedure we use. We also run a Gatekeeper-tuned small model if compute allows, since confidence tuning directly attacks the "confident but wrong" failure our method claims to fix. The second strong competitor is a length-aware router: the same classifier and the same calibration, but given only $L$, task type, and confidence. It also gets the three-way action space. This is the honest test, because LaRA's tipping point suggests that length alone might handle most of the retrieval-versus-full-context decision.
Fairness rules apply to every system. All use the same small and large models, the same retriever and chunking, and the same calibration and test splits. Every learned router gets an identical hyperparameter search budget, and self-consistency samples are charged to the cost of that baseline. As the small model we use Llama-3.2-3B, the model on which LaRA saw the largest retrieval effect. As the large model we use a model with an equal context window, such as Qwen2.5-14B-Instruct, which 100-LongBench evaluates.
The key ablation isolates dispersion. We keep coverage $C$ but replace $K$ and $D$ with values computed from randomly permuted match positions. If permuting positions costs nothing, the "where it looked" claim is false and our method reduces to a lexical grounding check.
For reference, the file contains no cascade results on long-context benchmarks: no reported result found on Lacuna for cascade routing on LongBench, LaRA, or 100-LongBench. The closest numbers come from other regimes. RouteLLM reports 50% APGR (average performance gap recovered) while calling the strong model about 13% of the time on MT-Bench. C3PO reports 62.5% accuracy at \$0.0019 per question on MATH-500. We will therefore reproduce the confidence and length baselines ourselves. Beating them means a strictly higher accuracy-versus-cost curve, with at least a 3-point accuracy gain at a matched large-model call rate on the long bins (32k and above), and a measurable reduction in harmful escalations.
Why not just…?
Why not just threshold the small model's confidence? This is the main threat. Gatekeeper shows that confidence tuning can sharply improve deferral. The evidence that confidence misses dependency failures is indirect: A Tale of Two Structures shows that instruction tuning and greedy decoding distort a model's perplexity signature. Confidence also says nothing about whether the large model will fail. The experiment settles the question directly. We compare failure-prediction AUROC (area under the ROC curve) for confidence alone, footprint alone, and both combined, separately in each length bin. If confidence matches the footprint on the small-model-error head, we will report the footprint's value as residing only in the dominance-violation head.
Why not just route on input length? LaRA's tipping point makes length a strong proxy for when to prefer retrieval. Our 100-LongBench analysis at fixed length is designed precisely to show whether footprint dispersion adds predictive power at a fixed $L$. If it adds nothing, a length rule suffices and we will say so.
Why not just always use retrieval with a cheap model? LaRA and LongBench both report that retrieval fails on multi-hop and comparative questions whose evidence is scattered, and that it hurts models with strong native long context. Our "always retrieved" baselines measure this cost directly, and the per-task breakdown shows where full context remains necessary.
Why not just use an LLM judge or speculative decoding? A judge model adds a second neural call per query; Cascadia's GPT-4o judge adds about 0.27 s. Speculative decoding with n-gram drafts is lossless but runs the large model on every query, so it cannot save the calls that cascades avoid. We include an entailment-model verifier as a cost-charged reference to quantify the first trade-off.
Proposed Experiments
We evaluate on the question-answering subsets of LongBench (single- and multi-document), the full LaRA suite of 2,326 cases across novels, papers, and financial statements, and length-controlled 100-LongBench. We exclude summarization from the main claim because paraphrased outputs weaken exact matching, and we report it separately as a stress test.
The metrics are accuracy-versus-cost curves, where cost is measured as large-model tokens and as wall-clock time including footprint computation. We also report APGR at matched call rates, AUROC for both classifier heads, and the dominance-violation rate: the fraction of escalations where the large model with full context is wrong and the small model or retrieved-context variant was right.
Three decisive checks determine the outcome. The first asks whether $D$ and $K$ predict errors at fixed length on 100-LongBench. The second asks whether permuting match positions removes the gain. The third asks whether the three-way policy cuts dominance violations in the 128k bin relative to the length-aware router. If all three hold, footprints become a cheap, API-compatible routing signal for long inputs. If only coverage matters, the result is still useful: lexical grounding alone would suffice for long-context deferral, and a much simpler system would do.
References
- RouteLLM: Learning to Route LLMs from Preference Data
- Gatekeeper: Improving Model Cascades Through Confidence Tuning
- C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning
- Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning
- Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
- Faster In-Context Learning for LLMs via N-Gram Trie Speculative Decoding
- Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing
- 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
- L2M: Mutual Information Scaling Law for Long-Context Language Modeling
- Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning
- A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language?