Retriever-Weighted Attention: Passing Neighbor Distances Into the Reader

A retrieval-augmented language model works in two steps. A search system finds the passages closest to a question, measured by distances in an embedding space, and a large language model then reads those passages and writes an answer. The two steps disagree about what matters. The search system knows that passage 1 was a near-exact match and passage 17 barely qualified. The language model never sees that distinction. Every passage arrives as plain text in one long prompt, and the model spreads its attention according to where text sits in the prompt rather than how relevant it is. A well-documented result is that answers buried in the middle of the prompt are often missed. For a smart reader, the proposal in two sentences is this. Old nearest-neighbor methods solved the same problem decades ago by weighting each retrieved neighbor according to its distance, so that far neighbors cannot outvote near ones. We bring that rule into the reader by adding the logarithm of each passage's neighbor weight to the attention scores for that passage's tokens, which scales the attention the passage receives by exactly that weight.

The reason to expect this to matter is a specific result from vision. In large image memories, plain nearest-neighbor voting gets worse as more neighbors are retrieved, because distant ones swamp the signal. Weighting neighbors by $1/(\alpha+\text{rank})$ stops that decline and survives 60% corrupted labels (Towards flexible perception with visual memory). Retrieval-augmented generation shows the same symptom. Accuracy rises and then falls as passages are added (Beyond RAG vs. Long-Context), so practitioners must hand-tune how many passages to include. If distance weighting carries over, that tuning step disappears at zero extra inference cost.

Core Claim

Attention is a softmax, so adding a constant $\log w_i$ to the attention score of every key token in passage $i$ multiplies the total attention mass that passage receives by $w_i$, before the scores are renormalized. Concretely, for a query token $t$ (a question or answer token) and a key token $j$ belonging to passage $i(j)$, we use

$$\tilde{s}_{tj} = s_{tj} + \beta \log w_{i(j)},$$

where $s_{tj}$ is the model's original attention score and $\beta \ge 0$ is a single global strength. Attention among tokens inside the same passage is left unchanged.

The neighbor weight $w_i$ comes from the nearest-neighbor literature. The first option is the rank-harmonic weight $w_i = 1/(\alpha + r_i)$, where $r_i$ is the passage's retrieval rank. The second is a per-query normalized kernel

$$w_i = \exp\!\left(-\frac{d_i - d_1}{\tau_q}\right),$$

where $d_i$ is the retriever distance of passage $i$, $d_1$ is the distance of the closest passage, and $\tau_q$ is the spread (interquartile range) of the top-$k$ distances for that query. Setting $\tau_q$ per query keeps the kernel meaningful even though raw cosine scores change scale from query to query.

When $w_i$ takes the inverse-distance form $d_i^{-p}$, this bias reproduces the negative-log distance score from Inverse distance weighting attention. That paper showed the score recovers Shepard interpolation. It did so on learned prototypes in small classifiers. Here the score is applied at inference, without training, to the passages a retriever returns.

The claim is falsifiable on three counts. First, with $\beta$ and $\alpha$ fixed once on a development set, accuracy should stay roughly flat as the number of retrieved passages $k$ grows from 5 to 50, where unweighted prompting declines. Second, it should match or beat hard top-$k$ truncation with the best $k$ chosen per dataset. Third, it should lose its benefit when the weights are randomly reassigned across passages, which shows that the gain comes from the retriever's signal and not from simply damping the context overall.

Relation to Prior Work

Two mechanistic papers motivate the gap and set the bar.

Found in the Middle models document attention as relevance plus a positional bias. It estimates the bias from dummy documents and subtracts it, gaining 6–15 points in hard middle-position settings. Its relevance signal is read off the model's own attention.

PINE makes attention between documents bidirectional and reassigns position indices according to internal attention importance. It costs 2–8x wall time.

Both methods ignore the retriever's distances. We do the opposite: we import an external geometric signal and leave positions alone. The two signals could be complementary, and we test whether combining them helps.

LDAR supplies the closest idea at the passage level. It reads the shape of the similarity-score distribution with a policy trained by reinforcement learning and keeps a contiguous band of passages. Once the band is chosen it concatenates the passages unweighted, and its authors leave the weighting inside the context open. Our method is the soft, training-free counterpart: a band is a 0/1 weight, and we test whether continuous weights do better.

The nearest-neighbor language-model line supplies the template for adapting trust to distance. kNN-LM mixes a distance-weighted neighbor distribution with the model's own predictions using a fixed coefficient. Adaptive kNN-LM buckets that coefficient by the distance to the single nearest neighbor and improves Wikitext-103 perplexity from 16.12 to 15.50. Both work on individual tokens. Our $\tau_q$ extends their one-neighbor heuristic to the whole neighborhood, and it acts on passages in a prompt rather than on next-token mixtures.

RbFT constrains the design. It shows that untuned models trust defective passages blindly and fixes this with LoRA fine-tuning. We ask how much of that robustness a free reweighting recovers.

One caution comes from the same inverse-distance paper. Its distance scores trailed dot-product attention on MNIST (88.20% vs 93.15%). That is why we add a bias on top of the model's native scores rather than replace them.

Research Program

We implement the bias as an additive per-key-position mask in open-weight readers: Llama-3.1-8B-Instruct and one larger model of around 70B. The bias works with fused attention kernels that accept additive masks.

Distances come from two sources. The first is a BGE dense retriever. The second is a cross-encoder reranker score converted to a distance, which tests whether better-calibrated relevance makes the weighting more useful.

We sweep four design choices:

  • Weight family: rank-harmonic, normalized kernel, or inverse power.
  • Strength $\beta$.
  • Which layers receive the bias: all layers, or only the middle third.
  • Which tokens receive it: question tokens only, or question tokens plus generated tokens.

Hyperparameters are tuned on a held-out split of Natural Questions only and then frozen for all other datasets. This guards against the method winning through per-dataset tuning.

Baselines

The naive reference is flat retrieval-augmented prompting. We take the top-$k$ passages in retriever order and concatenate them, run at every $k$ in the sweep. Closed-book answering is reported as the floor.

The strongest simple competitor is hard truncation with the best per-dataset $k$, plus the same passages reordered by score so the best passages sit at the start and end of the prompt. This is what a careful practitioner would actually deploy. It shares our retriever, reader, and decoding. It gets a larger tuning budget than we do, because $k$ is chosen on each test distribution's development split. Beating it means higher accuracy at matched or fewer prompt tokens, with no per-dataset tuning on our side.

The strongest existing mechanistic method is Found in the Middle. We reimplement it on the same reader and retrieved sets and also report it combined with our bias. The file gives only relative gains of 6–15 points for it and no absolute accuracy on Natural Questions for an open model, so no reported result was found on Lacuna for its absolute accuracy in our exact setting, and we will measure it ourselves.

PINE is run with wall-time reported alongside accuracy. Beating it means matching its accuracy within a point at roughly 1x latency instead of 2–8x.

LDAR is included where we can retrain its band selector on our retriever's scores. The file reports that LDAR uses 25–63% of full-context tokens, but no reported result was found on Lacuna for its absolute accuracy on these benchmarks.

Three ablations isolate the key ingredient, which is the retriever signal:

  1. Shuffled weights: the same multiset of $w_i$, randomly permuted across passages.
  2. Uniform damping: every passage gets the same average bias.
  3. Position-only weights: weights that depend on prompt position rather than retrieval rank, decoupled by shuffling passage order before applying rank weights.

Why not just…?

Why not just truncate to the best top-$k$ and sort by score? This is the most serious threat, because hard truncation is a special case of our weights, with $\log w = -\infty$ beyond $k$. If a single tuned $k$ matches our soft weighting on every dataset, the contribution shrinks to "removes one hyperparameter." We would report that honestly. The case for soft weights rests on question sets where the gold passage's rank varies widely. In those sets any fixed cutoff either drops gold passages or keeps distractors, as in the multi-hop second-hop recall drop of up to 18% that ChainRAG reports. The accuracy-versus-$k$ curves and per-question gold-rank breakdowns settle this.

Why not just reorder passages so the best ones sit at the edges? Found in the Middle reports that positional bias persists under reranking and reordering, and that calibration beats reranking strategies by up to 15 points in Recall@3. Reordering also cannot reduce how much attention distractors receive in total. Our reordered-truncation baseline tests this directly.

Why not just write the relevance scores into the prompt as text? It costs nothing and needs no access to the model's attention, so we include it as a baseline: each passage is tagged with its rank and score. We expect it to help little, because nothing makes the model act on the numbers. If it matches our method, that would argue for the text route, which is more portable.

Why not just fine-tune for robustness, as RbFT does? Fine-tuning changes the model and needs synthetic defect data. Our method is free at test time, and the two can be stacked. We run the Normal-tier condition, where 40% of passages are defective, to measure how much of the robustness gap reweighting closes. The file gives no raw RbFT accuracies to compare against.

Proposed Experiments and Decisive Checks

The primary testbed is the multi-document Natural Questions setup from Lost in the Middle, with 10, 20, and 30 documents and the gold passage placed at controlled positions. That paper reports roughly 75% accuracy for GPT-3.5-Turbo with the gold document first and about 53% in the middle at 20 documents. Our open readers will be re-measured under the same protocol.

SynthWiki removes help from the model's memorized knowledge. HotpotQA, MuSiQue, and 2WikiMultiHopQA test multi-hop questions, where the gold passages are not all top-ranked. Natural Questions with 40% injected defective passages, following RbFT's Normal tier, tests robustness.

Metrics are answer accuracy (exact match or answer containment) and F1, the spread between best and worst gold position, accuracy as a function of $k$, and tokens and wall time per query.

Four outcomes are decisive:

  1. Flatness: a near-flat accuracy-versus-$k$ curve from 5 to 50 passages, against a declining flat-prompting curve.
  2. Truncation: a tie or win against per-dataset-tuned hard truncation, with frozen hyperparameters.
  3. Signal: a clear collapse under the shuffled-weight ablation.
  4. Position bias: a reduced gold-position spread at 1x latency, compared with PINE.

If the second check fails, we narrow the claim to the regime where it does hold, namely wide gold-rank variance and multi-hop retrieval. If the third check fails, the gain comes from damping the context in general rather than from the retriever's distances, and the idea is wrong.