Search systems that feed language models usually run in two stages. A fast first stage retrieves candidate passages using compressed embeddings. A slow second stage reranks them with a model that reads query and passage together. The fast stage is known to favor passages whose relevant content appears near the start, so the slow stage often overturns its ranking exactly when the answer sits late in a passage. Distributed databases faced a similar shape of problem and solved part of it with speculation. They hand the application a fast, possibly wrong preliminary answer, let it start work immediately, and correct it when the authoritative answer arrives. We propose to bring that idea to retrieval-augmented generation. The language model starts generating on the fast retriever's results while the reranker runs. A cheap gate decides, per query, whether that early start is likely to survive. The gate's main input is where in the passage the matching evidence lies.
The claim is about where the fast stage fails. Its errors are not random staleness. They are concentrated, predictably, on queries whose evidence is positioned late, and that concentration can be observed cheaply before the reranker finishes. If this holds, a position-aware gate should recover most of the reranker's accuracy at close to the fast stage's latency. Naive speculation, by contrast, is actively harmful under lead bias: the preliminary answers are sometimes adversarially wrong, not slightly stale.
Background and the gap
The method comes from Incremental Consistency Guarantees for Replicated Objects. It introduced "Correctables," a promise-like object that delivers a fast, weakly consistent value and later a final, strongly consistent one. Applications can speculate on the first value. In multi-region deployments this cut strong-consistency latency by 40% (100ms to 60ms in an ad-serving workload), at the cost of a 6% drop in throughput and a 10% increase in bandwidth. That work assumes preliminary values are usually correct and that their errors are unstructured. Our setting breaks both assumptions.
Two empirical studies supply the motivation. An Empirical Study of Position Bias in Modern Information Retrieval built position-bucketed retrieval benchmarks and measured how sensitive each model is to answer position, using a Position Sensitivity Index (PSI). Dense and late-interaction models lose on average 15.6% when relevant content appears late; jina-colbert-v2 reaches PSI 0.499 on its long-passage benchmark. Full-interaction rerankers stay below 0.03; bge-reranker-v2-m3 holds NDCG@10 between 93.53 and 96.10 across all positions. Collapse of Dense Retrievers stress-tested retrievers with controlled foils. It found that Dragon+ and Contriever prefer short, early, or literally matching foils over factual evidence more than 90% of the time, and that feeding the preferred foil to GPT-4o costs 34% accuracy.
Together these two results say the fast stage's disagreements with the slow stage are large and structured by position. That is precisely the condition under which ungated speculation backfires. Nothing in the file connects the two lines of work.
The training-side debiasing work constrains the design rather than supplying it. Demoting the Lead Bias via Alternating Adversarial Learning trains a summarizer to hide position from an adversary. Countering the Effects of Lead Bias via Multi-Stage Training and Auxiliary Losses pretrains on shuffled documents and adds a loss that aligns sentence scores with their ROUGE overlap. Both reduce bias, but neither removes it: extractive models on late-content documents still struggle to exceed ROUGE ~21 against an oracle of 42.21. We therefore treat residual lead bias in any fast model as a given to be managed at inference time, not trained away.
A looser conceptual parallel is The Homeostasis Protocol. It lets nodes act without coordinating while a locally checkable condition guarantees safety. Our gate plays the role of that local check, but it is learned and probabilistic, not proven. We do not claim the impossibility results of Distributed Transactional Systems Cannot Be Fast transfer formally to retrieval.
Core Claim
Let $T_r$ be reranker latency and $T_g$ be generation latency. Let $p$ be the probability that the reranker's top-$k$ set confirms the fast stage's top-$k$ closely enough that generation need not restart. Blocking costs $T_r + T_g$. Speculating costs, in expectation,
$$L_{spec} = p\,\max(T_r, T_g) + (1-p)\,(T_r + T_g),$$
which saves $p\cdot\min(T_r, T_g)$ but wastes about $(1-p)$ generations' worth of compute. The formula follows directly from the two outcomes of a speculation.
Lead bias drives $p$ down unevenly. It is high for early-evidence queries and low for late-evidence ones. A gate that speculates only where predicted $p$ is high keeps most of the latency savings while bounding both wasted compute and the risk of answering from a foil. The contribution is showing that the cheap features available before reranking predict $p$ well. The most important of these is the position of the best query-matching span within each retrieved passage.
Research Program
We would build a speculative retrieval-augmented generation pipeline in three parts. First, the fast stage retrieves top-$k$ passages. Second, generation starts immediately on that set while the reranker scores the top-$N$ candidates. Third, when the reranker returns, the system either confirms the speculative answer or restarts generation on the reranked set. We consider confirmation, version one, as unchanged top-1 plus top-$k$ overlap above a threshold, or, version two, as the speculative answer being judged equivalent to a non-speculative answer on a validation split. We will report both.
The gate is a small calibrated classifier, logistic regression or a two-layer network. It is trained on held-out queries to predict confirmation, using features that cost almost nothing at query time. The first feature is the normalized offset, within each top-$k$ passage, of the highest lexical-overlap window with the query, computed BM25-style. The remaining features are the dense score margin between ranks, rank disagreement between dense and BM25 for the same candidates, and passage length; the last targets the brevity bias reported by Collapse of Dense Retrievers. The threshold is set to meet a target error rate on answers that were speculatively produced and later confirmed. When the gate declines, the system waits for the reranker, so the worst case is the blocking pipeline.
The analysis has three aims. First, we will measure $p$ as a function of evidence-position bucket for several fast retrievers. Second, we will test whether the position feature carries predictive information beyond the retriever's own score margin. Third, we will trace the latency–accuracy–compute frontier as the gate threshold varies.
Baselines
The naive reference point is blocking: always wait for the reranker, then generate. This is bge-reranker-v2-m3 over dense candidates, which An Empirical Study of Position Bias reports at PSI 0.019 with NDCG@10 of 93.53–96.10 on the long-passage benchmark. It sets the accuracy ceiling our method must approach. The bar for success is staying within one NDCG@10 point and one point of downstream answer accuracy of it while cutting median and 95th-percentile end-to-end latency. The file reports no latency figures for this reranker (no reported result found on Lacuna for reranker latency), so we will measure latency ourselves on identical hardware.
The second reference is no reranking: generate directly from the dense retriever. This is the latency floor and the accuracy floor. It inherits the 15.6% late-position drop and, on the foil benchmark, the >90% foil preference.
The third is ungated speculation, the direct Correctables transplant from Incremental Consistency Guarantees: always speculate, restart on disagreement. It has the best possible latency when confirmations are frequent. We expect it to waste the most compute and, in any variant that emits early tokens to users, to expose foil-based answers. Beating it means matching its latency on early-evidence buckets while using substantially less wasted generation compute overall.
The key ablation is a margin-only gate: the same classifier with the position and length features removed. It isolates whether positional information matters at all. If the margin-only gate matches the full gate's frontier, the lead-bias framing adds nothing and we will say so.
All gated variants share the same retriever, reranker, generator, candidate depth, hardware, and the same held-out split for threshold tuning. Every system gets the same tuning budget, with only the gate threshold swept.
Why not just…?
Why not just use BM25, or a BM25–dense hybrid, as the fast stage? BM25 is robust to position, as An Empirical Study of Position Bias confirms. But Collapse of Dense Retrievers shows literal matching is its own shortcut that foils exploit. This is a real threat: a hybrid fast stage might raise $p$ enough that a gate is unnecessary. We therefore run the full pipeline with dense, BM25, and hybrid fast stages. If the hybrid with ungated speculation matches the gated dense pipeline, we report the hybrid as the simpler recommendation.
Why not just split passages into short chunks so late content becomes early? Chunking attacks within-passage position directly. It also produces shorter units, which the brevity bias already rewards, and it discards context the reader needs. This too is a real competitor. We include chunked dense retrieval (128- and 256-token windows) as a fast-stage variant and measure both its confirmation rate and its blocking accuracy on the long-passage benchmark. The gate is agnostic to the fast stage, so if chunking raises $p$ across the board the gate should simply speculate more often. Only if chunking reaches reranker accuracy unaided does our method lose its point.
Why not just debias the fast retriever with shuffling or adversarial training? The summarization evidence says such training narrows but does not close the late-content gap (ROUGE ~21 against an oracle of 42.21 in Countering the Effects of Lead Bias). It also costs retraining for every retriever. Where a debiased retriever exists, it is just another fast stage to plug in, and its residual bias is exactly what the gate is meant to catch.
Proposed Experiments
The retrieval-level study uses the position-bucketed SQuAD benchmark and the long-passage FineWeb-edu benchmark from An Empirical Study of Position Bias, built on FineWeb. For Contriever, Dragon+, jina-colbert-v2, BM25, hybrid, and chunked fast stages, we report confirmation rate against bge-reranker-v2-m3 by position bucket. Our first prediction is that confirmation falls monotonically with answer position for dense and late-interaction models and stays flat for BM25. If confirmation is flat across buckets for the dense models, the core premise fails and the project stops there.
The downstream study adds an open generator, so that compute is controlled, to answer questions on the SQuAD-derived split and on the ColDeR foil sets from Collapse of Dense Retrievers. We report answer accuracy, median and 95th-percentile end-to-end latency, the fraction of generations wasted, and the rate of foil-grounded answers that slip through confirmation.
The decisive checks are three. The full gate must dominate the margin-only gate on the accuracy–latency frontier, specifically on late buckets and foil sets. Gated speculation must stay within one point of blocking accuracy while recovering a substantial fraction of the latency gap to the no-reranking floor. And ungated speculation must show measurably higher waste and foil exposure on late-evidence queries. The last check reproduces, in retrieval, the failure of the "preliminary is usually right" assumption that Correctables relies on.
References
- Incremental Consistency Guarantees for Replicated Objects
- An Empirical Study of Position Bias in Modern Information Retrieval
- Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
- Demoting the Lead Bias in News Summarization via Alternating Adversarial Learning
- Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses
- The Homeostasis Protocol: Avoiding Transaction Coordination Through Program Analysis
- Distributed Transactional Systems Cannot Be Fast