Retrieval-augmented generation (RAG) means fetching passages from a document collection and having a language model answer with them in view. It faces a cost-versus-fidelity conflict that current systems settle by giving up one side. Context compressors shrink each retrieved passage into a few summary positions called anchors and let the model read only those. This cuts memory and latency sharply, but the exact rare tokens the answer needs, such as names, numbers and technical terms, are no longer directly visible. Copy-augmented readers take the opposite route. A pointer-generator is a decoder that, at each step, either generates a word from its vocabulary or copies a word from the input by pointing at it with attention. These readers keep every raw token visible so they can copy exactly, and they pay for that by attending over everything.
We propose a decoder that does both. It reads compressed anchors to decide where in the retrieved material to look. It then temporarily opens only the chosen segment's raw tokens to decide what to copy. The second claim is the part that makes anchor selection matter: how the context is cut into anchor segments determines whether this two-level copy can work. Uniform striding, which places an anchor every N tokens, often splits answer entities across segment boundaries. Placing anchors to cover entity-like spans should recover most of the copy accuracy of a fully uncompressed pointer at a fraction of the decoding cost.
Core Claim
The literature contains a direct contradiction. Anchor-based Large Language Models (AnLLM) compresses each segment's attention memory into anchor tokens and discards the rest. It reports a 99% reduction in the stored attention memory (the KV cache) and a 3.5x speedup for about a 1.5% average accuracy drop. Because the non-anchor tokens are thrown away, pointing back at source words becomes impossible. Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors (SAC) uses regularly spaced context tokens as anchors and reaches 15x compression with up to 23.5% F1 over ICAE. Generation in SAC still comes purely from the vocabulary. Meanwhile A Copy-Augmented Generative Model for Open-Domain Question Answering (FiD-PGN) shows that a copy gate built from existing cross-attention gives large gains on answers never seen in training. That gate needs flat attention over every token of every passage. LargePiG gets a training-free pointer out of decoder-only models, but it also points over the full prompt.
Our claim is that the copy distribution can be factored through anchors without losing fidelity. Write $\alpha_j$ for the decoder's attention over anchor segments $j$, and $\beta_{i \mid j}$ for a local attention over raw tokens $i$ inside segment $j$. The output distribution is then
$$P(w) = p_{gen}\,P_{vocab}(w) + (1-p_{gen}) \sum_{j \in \mathrm{top}\text{-}m(\alpha)} \alpha_j \sum_{i \in S_j,\; w_i = w} \beta_{i \mid j},$$
where $S_j$ is the set of raw tokens assigned to anchor $j$, $m$ is a small number of segments opened per step (we expect 1 to 4 to suffice), $P_{vocab}$ is the ordinary next-word distribution, and $p_{gen}$ is the usual learned gate. Ordinary decoding attends only to anchors. Raw tokens enter only through the inner sum, and only for the $m$ opened segments.
Factoring the copy this way makes the segmentation $\{S_j\}$ part of the copy mechanism, not just a compression detail. If an answer span is split across two segments, both must be opened and both must be ranked highly by $\alpha$. The anchor-selection question in this setting is therefore concrete: choose segment boundaries and anchor positions so that copyable spans are whole and each one sits under a single anchor.
Research Program
We build the method first in Fusion-in-Decoder (FiD), the T5 reader that encodes each retrieved passage separately and lets the decoder attend across all of them. Using FiD lets us compare directly with FiD-PGN's reported numbers. We then port it to a decoder-only model (LLaMA-3.1-8B) with SAC-style anchors.
In FiD, each passage's encoder output is pooled into anchor vectors, and the decoder cross-attends only to these. The raw encoder states are kept off the accelerator, either in host memory or recomputed on demand per segment. They are fetched only for the top-$m$ segments chosen at each decoding step. The second-level attention $\beta$ is a single lightweight head that reuses the decoder query. This keeps the added parameter count in the same spirit as FiD-PGN's 1,537 extra parameters.
We compare four anchor-placement strategies under a matched anchor budget. The first is uniform striding as in SAC. The second is punctuation boundaries as in AnLLM. The third is entity-aligned segmentation, where boundaries are never placed inside spans tagged by an off-the-shelf named-entity and number tagger. The fourth is a greedy coverage strategy. Borrowing the coverage idea that NT-LLM uses for graph nodes, it places anchors to maximize the number of candidate spans fully contained in a segment. It does this while respecting a maximum segment length, so that NT-LLM's sensitivity to the coverage radius is controlled explicitly and not tuned by hand.
The analysis targets two failure modes the file documents. The first is "lazy copying," described for Concept Pointer Network for Abstractive Summarization, where the model copies when it should synthesize. We track this through the rate at which $p_{gen}$ chooses copying on questions whose gold answers are not present in the passages. The second is attention dispersal over long contexts. We test whether the two-level factorization stays sharp as the number of passages grows from 25 to 100.
Baselines
The naive reference point is the same FiD-T5-base reader with plain vocabulary generation over uncompressed passages. FiD-PGN reports 48.5 EM on NaturalQuestions for this reader with 100 passages, and we rerun it under our retrieval to confirm that figure.
The strongest existing method is FiD-PGN itself. It is the closest prior work and it copies over all raw tokens, so it is the accuracy ceiling our factorization tries to approach. It reports 51.4 EM on NaturalQuestions with 25 passages, plus a 9.6-point gain on the No Overlap subset (answers not seen in training) A Copy-Augmented Generative Model for Open-Domain Question Answering. We retrain it with identical retrieval, passage counts, T5-base initialization, step budget and learning-rate sweep.
The compression competitor is a SAC-style FiD. It uses the same anchors and the same budget, decodes over anchors only, and has no copy channel. This isolates what pointing through anchors adds.
For the decoder-only port, the competitors are LargePiG over the full prompt and SETR set selection (Shifting from Ranking to Set Selection for Retrieval Augmented Generation) followed by ordinary generation. SETR reports 47.14% accuracy on MultiHopRAG with about 1,240 tokens, against 45.69% for RankGPT. All decoder-only systems use the same 8B backbone and the same top-20 candidate pool.
The key ablations hold the model fixed and vary the anchor-placement strategy. Separately, they replace the two-level pointer with a one-level pointer over anchors only, which copies the anchor token itself.
Beating the field means three things on the NaturalQuestions No Overlap subset and on rare-entity answers:
- falling within one EM point of retrained FiD-PGN at 25 passages,
- exceeding SAC-style FiD by a clear margin,
- doing both while cutting decoder cross-attention memory by at least 5x.
At 100 passages we expect to beat FiD-PGN outright, because there its flat attention should disperse.
Why not just…?
Why not just feed everything to a long-context model and let it copy natively? For strong generators this is a real threat. How Does Knowledge Selection Help Retrieval Augmented Generation? finds that, for capable models, providing all retrieved knowledge is a robust baseline and that recall matters more than precision. That finding cuts in our favor on one point: it argues for keeping many passages, and compression is what makes many passages affordable. The same paper also finds that distractors on HotpotQA push accuracy below the no-retrieval baseline, which a model attending over everything is exposed to. We run full-context LargePiG and full-context plain decoding at matched passage counts. If they match our accuracy, our contribution reduces to an efficiency result, and we will report it as such.
Why not just select three good passages with SETR and copy from them uncompressed? If set selection reliably finds the evidence, the context is already small and compression has little left to do. SETR itself names the weakness: selection cannot recover evidence missing from the candidate pool, and it commits before generation starts. Our method instead defers selection into every decoding step via $\alpha$. We sweep the pool size and measure where SETR-then-copy and anchor-pointing cross over. If SETR-then-copy wins at every pool size, the proposal fails on decoder-only models and we will say so.
Why not just keep anchors and copy the anchor token itself? This is the one-level ablation. Under uniform striding, anchor positions rarely coincide with answer tokens, and multi-token entities need the raw neighbors. We expect this ablation to collapse on rare entities. If it does not, the second level is unnecessary.
Why not just extract a span with a joint extractor? A single-span extractor in the style of A BERT Baseline for the Natural Questions, which reports 52.1 short-answer F1, cannot compose answers drawn from disjoint segments. Our HotpotQA evaluation tests exactly this composition.
Proposed Experiments
The primary benchmarks are NaturalQuestions and TriviaQA in the open-domain setting, matching FiD-PGN's protocol. No reported TriviaQA EM value for FiD-PGN was found on Lacuna, so we report our own reruns. Multi-hop evaluation uses HotpotQA in the distractor and full-wiki settings (HotpotQA) and MultiHopRAG. Metrics are:
- EM and answer F1,
- EM restricted to answers containing numbers or rare entities,
- the No Overlap split,
- decoder memory, latency, and the number of segments opened per answer.
The decisive check is a boundary analysis on NaturalQuestions. Under uniform striding, we flag test answers whose gold span crosses a segment boundary. The method is worth pursuing only if two things hold. First, copy accuracy on those answers drops sharply relative to answers that do not cross a boundary. Second, entity-aligned or coverage-based anchors close most of that gap while the anchor budget stays fixed.
The second decisive check scales passages from 25 to 100. FiD-PGN's accuracy per added passage should flatten as its attention spreads thin. The two-level pointer, which opens only $m$ segments, should keep improving. If neither check holds, the clean conclusion is that anchor compression and exact copying are not in conflict at these scales, and that conclusion is itself worth publishing.
References
- Anchor-based Large Language Models
- Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors
- A Copy-Augmented Generative Model for Open-Domain Question Answering
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models
- Concept Pointer Network for Abstractive Summarization
- Shifting from Ranking to Set Selection for Retrieval Augmented Generation
- How Does Knowledge Selection Help Retrieval Augmented Generation?
- A BERT Baseline for the Natural Questions
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering