Retrieval-augmented generation (RAG) means letting a language model look up passages from a document collection before it answers. Its iterative form alternates the two steps: the model drafts an answer, uses that draft to search again, and revises. This works because even a wrong draft often contains the right "bridge" words. It also fails in a predictable way. Once the draft commits to a wrong intermediate entity, every new search returns passages near that entity, the model reads them as confirmation, and the loop locks in. ITER-RETGEN, which runs this loop, reports that most gains arrive within the first two iterations and that the process plateaus after two or three rounds. Optimization researchers have a name for this behavior: a local optimum, a state from which every small step looks no better.
This note proposes treating the plateau the way local search treats a local optimum. The system detects stagnation cheaply, from the retrieved documents themselves. It then applies one bounded jump to the retriever's query, pushing it away from the region of the corpus it has already searched. The jump is sized per question as the smallest move that reaches new evidence while staying close to the original question. Accumulated evidence is kept, and the new region is accepted only if it supports an answer better. In two sentences: iterative RAG gets stuck because its own drafts keep steering retrieval back to the same document cluster. We detect when that happens and push the query a calibrated distance away from everything it has already seen, rather than sampling more drafts or building a search tree.
Core Claim
The claim is that much of the residual error in iterative multi-hop RAG comes from retrieval basins rather than from reasoning capacity. A retrieval basin is a set of near-duplicate passages that the draft-then-search loop cannot leave. If that is true, a stagnation-triggered, magnitude-calibrated jump in the retriever's query space should recover the missing evidence more cheaply than the two standard sources of exploration. The first is sampling more diverse drafts. The second is explicit tree search such as GRAT, which runs Monte Carlo tree search over sub-question and retrieval actions with process rewards.
The new ingredients relative to the closest work are three:
- The trigger is a retrieval-novelty signal, not token confidence.
- The perturbation acts on the query embedding and is repelled from visited evidence, not taken from a fixed rewrite template.
- The jump size is chosen per question under an explicit lower bound on reaching new documents and upper bound on drifting from the question.
That last ingredient is the "Goldilocks" problem from local search, made operational.
Let $D_t$ be the top-$k$ passages retrieved at iteration $t$, and let $V_{t-1}$ be the union of all passages retrieved before. Novelty is
$$\nu_t = \frac{|D_t \setminus V_{t-1}|}{k},$$
the fraction of this round's passages that are new. A stall is declared when $\nu_t < \rho$ and the extracted answer has not changed since the previous round.
At a stall, let $e(\cdot)$ be the retriever's normalized embedding function, $x_t$ the current query text (the question plus the current draft), $q$ the original question, and $c_V$ the mean embedding of the passages in $V_{t-1}$. The jumped query is
$$\tilde q(\lambda) = \frac{e(x_t) - \lambda\, c_V}{\lVert e(x_t) - \lambda\, c_V \rVert},$$
with $\lambda$ chosen as the smallest value on a short grid that satisfies two conditions. Retrieval with $\tilde q(\lambda)$ must have novelty at least $\rho$, and $\cos(\tilde q(\lambda), e(q))$ must be at least $\kappa$. The first condition says "leave the basin." The second says "do not become a random restart."
Research Program
The method borrows its structure from three optimization papers, and each plays a different role:
- The repulsive direction. The Distributed Boosting Scheme in Distributed Non-convex Optimization of Multi-agent Systems Using Boosting Functions to Escape Local Optima turns a zero gradient at a trap into a non-zero "boosted" direction that pushes agents away from it. Subtracting the visited-evidence centroid is our analogue. Unlike that work, we have no differentiable objective, only retrieval sets.
- The scaling principle. The surrogate optimizer in Hospital simulation model optimisation with a random ReLU expansion surrogate model perturbs local minima with noise scaled to the parameter bounds and the problem's dimensionality. It reports beating Bayesian optimization under a 200-evaluation budget, with average best objective 14.81 against 16.78. We replace its fixed scaling rule with a feasibility search over $\lambda$, because embedding geometry differs from question to question.
- The control loop. DTNEST in Symbiotic Local Search for Small Decision Tree Policies in MDPs alternates cheap local improvement with a structural jump when local search stalls. We adopt that alternation, plus an iterated-local-search acceptance rule. After a jump, the loop runs up to two ordinary iterations from the new region. It keeps the new answer only if the answer's support score is higher than before the jump; otherwise it reverts, with the visited set still enlarged. The support score is the fraction of answer sentences that a small entailment model judges to be backed by retrieved passages.
Two adversarial papers both constrain and motivate the choice to perturb in embedding space rather than in tokens. Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models shows that a 5–10 token gradient-optimized prefix can steer a dense retriever to an arbitrary target passage. Evaluating RAG Robustness to Symbolic Perturbations shows that a single emoticon collapses query vectors into irrelevant clusters, with near-100% attack success. Together they say that query embeddings are easy to move, so a jump is cheap. They also say that token-level edits move embeddings by unpredictable and often destructive amounts. We therefore never perturb the text. We perturb the vector directly, under an explicit cosine bound, and we report how often the bound actually binds.
The analysis component comes first. We will measure how often ITER-RETGEN trajectories enter a stall, defined by the novelty rule above. Then, among stalled trajectories that end wrong, we will measure what fraction have a gold supporting paragraph that is never retrieved. If that fraction is small, the errors are reasoning errors, not basin errors, and the proposal's premise fails. That is a cheap, early, falsifiable check.
ChainRAG motivates a second diagnostic. It finds that second-hop Recall@2 drops by up to 18% when sub-questions contain vague references, and its deterministic rewrite chain propagates a wrong first-hop entity downstream. We will tag stalls by whether the draft contains a wrong bridge entity, and check whether jumps recover disproportionately in exactly those cases.
Baselines
Naive reference. The do-nothing reference is ITER-RETGEN run for a fixed three iterations, with the same generator, retriever, corpus and $k$ as our method. Its documented plateau is the phenomenon we claim to fix. We also report single-shot retrieval as the floor.
Tabu exclusion. This baseline continues ITER-RETGEN but removes all previously retrieved passages from each new top-$k$. It isolates whether the benefit comes simply from "not seeing the same documents," or from moving the query out of a cluster of near-duplicates. Beating it is the key ablation: our method must recover more gold paragraphs among stalled questions than tabu exclusion does.
Sampled self-consistency. This baseline runs $N$ independent ITER-RETGEN chains at temperature 0.7–1.0 and takes a majority-vote answer. $N$ is chosen so that the total number of generator calls and retrieval calls matches our method's average on each dataset. It is the strongest cheap exploration competitor.
Strongest existing method. The strongest competitor is GRAT's tree search, run without its self-training stage and with the same base model, at two budgets. One is matched to our compute. The other is GRAT's default budget, to see how much of the tree's gain a single trajectory can recover. For reference, GRAT reports 75.2% accuracy on 2WikiMultiHopQA with Llama3-70B, and its Llama3-8B rises from 60.6% to 66.8% after training on 11,000 trajectories (GRAT). These figures come from GRAT's own setup and are not directly comparable to ours, which is why we rerun it. Beating GRAT means matching its accuracy at a fraction of its LLM calls, or exceeding it at matched calls. Exceeding its full-budget accuracy is not expected.
Other reported reference points. FLARE, which triggers retrieval on low-confidence tokens, reports 51.0 exact match (EM) on 2WikiMultiHopQA with text-davinci-003. Stop-RAG reports 47.0 F1 on MuSiQue. We rerun FLARE with our generator.
Ablations of the method's own ingredients. Three ablations separate the trigger, the direction and the calibration:
- Untriggered: jump at every iteration.
- Fixed-magnitude: a single global $\lambda$ tuned on development data.
- Random-direction: isotropic noise of the same norm instead of repulsion from $c_V$.
All systems get the same development-set tuning budget, a fixed number of configurations, for their own hyperparameters.
Why not just…?
Why not just raise the temperature or sample more drafts? This is the most serious threat. If wrong drafts are the cause, diverse drafts might fix them. Our counter-hypothesis is that a high-temperature draft still conditions on the same retrieved passages and tends to rediscover the same entity. FLARE's finding that drafts strongly steer retrieval supports this: draft-based queries beat previous-sentence queries by nearly 10 EM points on 2WikiMultiHopQA. The cost-matched self-consistency baseline settles it. If its accuracy-versus-calls curve dominates ours, the proposal fails.
Why not just rewrite the query with diverse templates at the stall? DMQR-RAG shows that diversity in embedding space beats quantity of rewrites, reporting a 14.46% gain in precision at five retrieved passages on FreshQA. But it rewrites once, up front, with fixed templates that have no explicit relation to what has already been retrieved. We include a variant that applies DMQR-style rewrites at our stall trigger. If it matches the embedding jump, the honest conclusion is that the trigger, not the vector-space operator, carries the gain. We would then report that.
Why not just exclude already-seen documents? Tabu exclusion is the plain version of our idea, and it is run as a baseline. It removes exact passages but not their paraphrases or same-topic neighbors. Our method should therefore win specifically on corpora with dense near-duplicates, such as full Wikipedia for HotpotQA, and tie elsewhere. We will report results split by local corpus density.
Why not just let a learned controller or the model's self-critique decide? Stop-RAG's learned value function beats prompted self-stopping (47.0 vs 44.8 F1 on MuSiQue). That suggests self-assessment is a weak signal. Stop-RAG's action set is only stop or continue, though. A natural extension adds the jump as a third action, and we treat that as follow-up work, not a competitor.
Proposed Experiments
We will use HotpotQA (full-wiki setting), 2WikiMultiHopQA, MuSiQue-Ans and Bamboogle, the suite shared by ITER-RETGEN and GRAT. Retrieval is over a Wikipedia passage index with a self-hosted dense retriever, which gives white-box access to query vectors. The generators are Llama3-8B and Llama3-70B. We report EM and F1, gold-paragraph recall, number of LLM and retrieval calls, and accuracy-versus-calls curves.
The decisive checks are three:
- Basin premise. At least a substantial minority of wrong ITER-RETGEN answers must be stalled trajectories with unretrieved gold paragraphs.
- Recovery on stalled questions. On those questions, the calibrated repulsive jump must recover the gold paragraph more often than tabu exclusion, random-direction noise and template rewriting.
- Cost efficiency. At matched calls, overall accuracy must exceed sampled self-consistency, and the method must close a meaningful part of the gap to GRAT at a fraction of GRAT's calls.
A secondary check tests the Goldilocks claim directly. We will plot recovery and answer accuracy against $\lambda$ for stalled questions. We expect a curve that rises and then falls: small jumps fall back into the same cluster, and large ones behave like restarts. We also expect the per-question calibrated $\lambda$ to track the peak better than any single fixed value.
References
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- Distributed Non-convex Optimization of Multi-agent Systems Using Boosting Functions to Escape Local Optima
- Hospital simulation model optimisation with a random ReLU expansion surrogate model
- Symbiotic Local Search for Small Decision Tree Policies in MDPs
- Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
- Evaluating RAG Robustness to Symbolic Perturbations
- Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering
- Active Retrieval Augmented Generation
- Stop-RAG: Value-Based Retrieval Control for Iterative RAG
- DMQR-RAG: Diverse Multi-Query Rewriting in Retrieval-Augmented Generation