Teaching the Copy Gate Which Passages to Distrust

When a question-answering system answers a multi-step question, it usually retrieves some documents, reads them, pulls out an intermediate fact (a "bridge" entity such as the name of a director), and uses that fact to search again. A common way to keep the reader honest is a pointer-generator. This is a decoder that, at each output token, chooses between generating a word from its vocabulary and copying a word directly from the retrieved text. Copying keeps rare names exact and reduces invented facts. It has a blind side, though. Retrievers reliably return near-miss documents ("hard negatives") that share the question's entities but answer something else, and a copy mechanism will copy from those just as readily. One wrong copied bridge entity at the first step then sends every later search in the wrong direction.

This proposal moves an idea from search ranking into the copy mechanism itself. Re-rankers improve sharply when they are trained against the exact near-misses their own upstream retriever produces, rather than against random or mismatched negatives. We propose to train the copy gate the same way. Its copying mass should concentrate on gold passages and stay off the specific decoys that this pipeline, at this step, actually surfaces. We also construct the training negatives by running the real multi-step pipeline, so they include the documents that appear after an earlier step has already gone wrong. Two things follow if this works. The reader should copy fewer wrong bridge entities. And the distribution of where the copy mass lands should serve as a per-step warning that the previous retrieval failed, early enough to search again instead of drifting into a long run of useless queries.

Core Claim

The claim is that per-passage copy mass in a pointer-generator reader can be supervised with a contrastive loss over trajectory-matched retriever negatives. A reader trained this way lowers decoy copying and improves multi-hop answer accuracy beyond what one gets from training the same reader on the same negatives without the copy-level loss, or from filtering those negatives with a retriever-aligned re-ranker. The trained copy mass should also detect failed hops better than token log-probabilities, which copying is known to inflate.

The closest work leaves a specific opening. A Copy-Augmented Generative Model for Open-Domain Question Answering builds the pointer-generator we reuse (FiD-PGN). It takes the reader's existing cross-attention as the copy distribution and adds only 1,537 parameters to a 220M T5 model. It is trained on standard single-hop DPR passages with no negative-aware objective, and it is never tested on multi-hop chains. Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline supplies the training principle. Its Localized Contrastive Estimation (LCE) samples negatives from the target retriever's own top results and applies a group-wise softmax loss. It showed that a better first-stage retriever does not help unless the re-ranker is trained on that retriever's errors. The method only ever scores passages in a cross-encoder and never touches a generator. Rethinking the Reranker: Boundary-Aware Evidence Selection for Robust Retrieval-Augmented Generation (BAR-RAG) is the nearest three-way connection. It trains an evidence selector with reinforcement learning to feed a generator passages at the edge of its competence, then discards the selector at test time. That design has three costs: the generator still faces raw retriever noise at inference, it has no copy mechanism, and it requires expensive rollouts on both selector and generator. Our approach is supervised, keeps an explicit test-time signal, and acts on token-level copying.

Two diagnostic papers motivate the work. Beyond Confidence found "copy inflation": tokens copied from context get log-probabilities 0.50 nats higher on average, so answers lifted from a wrong document look confident. That paper's fix, Retrieval-Grounded Voting, scores finished rollouts by lexical overlap with retrieved text. By design it rewards the "well-grounded but wrong" case, which it reports in about 4% of cases. Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents shows the stakes. Cumulative retrieval recall correlates with accuracy at r = 0.99. Agents that retrieved gold evidence scored 73.8%, against 2.2% for those that did not. Failed trajectories run 2–3x longer, in a "wasted tail" of redundant queries. Detecting failure at the hop where it happens is therefore worth far more than repairing it at the end.

Research Program

The reader is FiD-PGN on T5-base and T5-large, placed inside an iterative two- to four-hop retrieve-read loop. At each hop the reader receives retrieved passages and decodes one of three things: a bridge entity, which becomes the next query; a final answer; or a special re-query token. For each decoded entity span, we define passage $j$'s copy mass as $C_j = \sum_{t} (1 - p_{gen,t}) \sum_{i \in j} a_{t,i}$. Here $t$ ranges over the span's decoding steps, $p_{gen,t}$ is the probability of generating rather than copying at step $t$, and $a_{t,i}$ is the attention weight on input token $i$, summed over the tokens of passage $j$. The new training term applies LCE's group-wise loss to these copy masses:

$$ \mathcal{L}_{copy} = -\log \frac{\exp(C_{+}/\tau)}{\sum_{j \in \{+\} \cup N_h} \exp(C_j/\tau)} $$

In this loss, $C_+$ is the copy mass on the gold passage for that hop, $N_h$ is the set of hard negatives at hop $h$, and $\tau$ is a temperature. It is added to the usual answer likelihood.

What sets this apart is how $N_h$ is built. We run the actual retriever, either BM25 or a dense retriever, through the actual pipeline. Hop-2 negatives are therefore drawn both from retrieval on the correct bridge entity and from retrieval on the reader's own wrong hop-1 outputs. This makes the training negatives match the inference distribution under error, which is the cascading condition that per-stage negative mining ignores. A second, smaller term covers the step-0 failure case. In a fraction of training hops we remove the gold passage. In those hops the target is the re-query token, with copy mass pushed toward zero on every passage. The model thus learns that "nothing here deserves copying" is an available output.

At inference, the re-query token, or a maximum per-passage copy mass below a threshold tuned on development data, triggers one alternate retrieval for that hop, with a hard cap on extra calls. We then repeat the core study on a decoder-only model, Qwen2.5-7B. Its attention over context serves as the copy distribution and a single learned gate plays the role of $p_{gen}$, following FiD-PGN's attention reuse. This tests whether the effect survives in the model class that matters today.

Baselines

The naive reference point is plain Fusion-in-Decoder. Next is FiD-PGN trained exactly as published, on standard top-retrieved passages, placed in the same multi-hop loop. On single-hop NQ, Liu et al. report 51.4 EM for FiD-PGN against 48.5 EM for FiD, and we reproduce that first to confirm our implementation. Every system uses the same retriever, the same passage budget per hop, the same hop cap, and the same backbone. Each gets an equal hyperparameter search of roughly twenty configurations on development data.

The strongest competitor on this question is a retriever-aligned re-ranker in front of an unmodified reader. Concretely, this is an LCE-trained cross-encoder (Gao et al.) trained on the same trajectory-matched negatives. It filters each hop's passages before standard FiD-PGN reads them, and its scores drive the same re-query trigger. It is the most dangerous rival because it uses identical negative information and the identical recovery rule, and differs only in where the discrimination happens. We give it extra compute, a separate encoder pass per passage, and report that cost. Beating it means higher answer F1 on MuSiQue-Ans at an equal or smaller number of retrieval calls, with a confidence interval from three seeds that excludes zero.

The ablation that isolates the key ingredient is the same reader trained on the same trajectory-matched negative contexts with answer likelihood only, with no $\mathcal{L}_{copy}$. A further ablation swaps trajectory-matched negatives for per-hop negatives mined from gold-query retrieval only, and another swaps in BM25 negatives behind a dense retriever. The last one replicates the mismatch effect from Gao et al. at the copy gate.

For published reference points, PATHFID+ reports 72.7 EM on HotpotQA distractor against 64.4 for FiD (Yavuz et al.). FrugalRAG reports 68.47% model-based evaluation on HotpotQA with 2.05 searches, against ReAct's 64.20% with 2.76 (FrugalRAG). For BAR-RAG the file gives only a 10.3% average improvement across benchmarks and a 12.5% gain over RAG supervised fine-tuning on Bamboogle, with no per-dataset scores. There is no reported model result on Lacuna for MuSiQue-Ans; the only figure is the human 78.0 answer F1 from MuSiQue. Our MuSiQue comparisons are therefore against baselines we run ourselves.

Why not just…?

Why not just put an LCE-trained re-ranker in front of the reader? This is the real threat, and the proposal is built so it can lose to it. Our argument is that a re-ranker scores passages without seeing what the reader actually uses. The "utilization gap" in Diagnosing Search Behavior, where evidence is retrieved but misused, is exactly a failure that passage scoring cannot see. If the re-ranker baseline matches us on MuSiQue-Ans F1 and decoy copy rate, we will report that the benefit comes from aligned negatives rather than from the copy gate. We would then recommend the simpler pipeline.

Why not just fine-tune an ordinary LLM reader on the same hard-negative contexts, without any copy gate? BAR-RAG suggests that curating which evidence a generator trains on already yields large gains. The same-negatives, no-copy-loss ablation tests this directly, run both on T5 and on Qwen2.5-7B. The copy gate earns its place only if it adds accuracy over that ablation, or if its per-passage mass predicts hop failure clearly better. Copy inflation in Beyond Confidence is the reason to expect the plain model's own probabilities to be a poor failure signal.

Why not just vote over rollouts with retrieval grounding, or retrieve more and search longer? Grounding-based voting works on finished outputs and, by construction, favors answers that echo a decoy. More search effort is largely uncorrelated with success, and failed trajectories already run 2–3x longer (Diagnosing Search Behavior). Even so, we include Retrieval-Grounded Voting over eight rollouts of the baseline reader, plus a doubled hop budget, as compute-matched comparisons.

Proposed Experiments

The main benchmark is MuSiQue-Ans. It was built so that single-hop shortcuts lose about 30 F1 points relative to HotpotQA (MuSiQue), and this matters because HotpotQA's shortcuts let independent sentence scoring do well (QUARK). We also report on HotpotQA (distractor and fullwiki) and on 2WikiMultiHopQA. MuSiQue-Full's unanswerable contrast questions test whether the re-query and "nothing to copy" behavior avoids becoming reckless abstention.

The metrics are answer EM and F1, supporting-passage F1, and decoy copy rate. Decoy copy rate is the fraction of predicted bridge or answer entities that appear only in non-gold passages. We also measure recovery rate, defined as accuracy on development questions where we deliberately remove the hop-1 gold passage from the first retrieval, along with retrieval calls per question.

Three checks decide the outcome. First, at matched retrieval calls, the full model must beat both the LCE re-ranker pipeline and the same-negatives ablation on MuSiQue-Ans F1, and must lower decoy copy rate. Second, when we inject hop-1 failures, per-passage copy mass must predict failure with higher AUROC than the reader's token log-probability and higher than the re-ranker's top score. That AUROC is what turns copy mass into a usable trigger, and its absence would sink the recovery story. Third, a retriever-swap test addresses the index-drift concern from the ranking literature. We train with negatives from BM25 and from the dense retriever, then evaluate behind each. If trajectory-matched copy training only transfers when the retriever matches, that is useful to know: it means the gate has to be retrained whenever the upstream index changes, just as re-rankers do.