Retrieval-augmented language models answer questions by reading passages that a search system has fetched for them. The passages that hurt them most are not random junk. They are near misses: text about the right topic that looks like an answer but is not one. Adding 18 such passages dropped Llama 2 7B-Chat from 56.42% to 37.81% accuracy on open-domain Natural Questions, while adding random documents actually helped (Rethinking Relevance). A common fix is a context filter, a small model that keeps only the useful sentences before the reader sees them. Today these filters learn from real benchmark questions and from the reader's own behavior. That supervision has two problems. It contains few clearly labeled near misses, and it ties the filter to one reader: compressors trained for one model family fail when paired with another (RECOMP).
This note proposes training the filter on generated data instead. We would take real Wikipedia text, generate questions from it, and plant near-miss sentences whose status is known by construction. One kind keeps the relation and swaps the subject ("Y was born in 1950" when the question asks about X). Another keeps the subject and swaps the relation ("X died in 1950"). These are mixed with topical filler and random noise at randomized densities and positions. The claim is that a filter trained this way, with no human QA labels and no reader in the training loop, holds up better as real retrieval noise grows. It should also transfer across reader families better than filters trained on reader feedback, because its labels come from how the text was built and not from any one model's preferences.
Core Claim
The randomization lesson from sim-to-real transfer ought to apply to text filtering. In robotics and vision, heavy procedural variation in synthetic training makes a model ignore features that don't matter (Domain Randomization). Diversity also matters more than realism (Diversity is Definitely Needed). Synthetic perturbation training for tool agents closed about 27% of the sim-to-real gap on failure types never seen in training (ToolRL-DR). Our specific bet concerns which variation matters. For text filters, the key variable is how close a distractor sits to the evidence, meaning whether it shares the subject or the relation. The amount of noise matters less. The obvious version of this idea is "add synthetic noise when training a compressor." It is weaker, because random and topical noise are the kinds readers already tolerate (Rethinking Relevance). Our version is a hypothesis about near-miss structure, and it can be falsified directly by the ablations below.
Relation to Prior Work
Some prior work motivates the gap. Cho et al. show that zero-shot reader accuracy falls as more retrieved documents are added (Improving Zero-shot Reader by Reducing Distractions). Cuconasu et al. isolate semantic distractors as the harmful kind (Rethinking Relevance). CogniLoad finds a non-monotone accuracy curve over distractor density in synthetic logic puzzles (CogniLoad). All three diagnose the problem, and none trains anything. We use their findings to design the training distribution, and we use CogniLoad's result as the reason to randomize density rather than fix it.
Other work supplies the method. RECOMP trains extractive and abstractive compressors that cut five documents to about 36 tokens with under 10% relative Exact Match loss, and it learns to output nothing when the context is useless. It also reports that abstractive compression becomes unfaithful on multi-hop HotpotQA and that its compressors fail to transfer across model families (RECOMP). We therefore use an extractive sentence selector with an abstain option, in the spirit of Min et al.'s finding that most questions need only one or two sentences (Efficient and Robust Question Answering from Minimal Context over Documents). QACG builds zero-shot fact-verification data by generating questions from evidence and swapping entities to make refuted claims. Its synthetic-only model held up on the adversarial FEVER-Symmetric set, dropping only from 78.1% to 77.1% (Zero-shot Fact Verification by Claim Generation). We reuse its question-generation and entity-substitution machinery. We apply it to multi-sentence retrieval contexts with controlled distractor density, which QACG never addresses.
A third group of papers are the direct competitors. CORE-RAG trains a compressor with reinforcement learning, rewarded by the reader's accuracy on real NQ and HotpotQA. It keeps 3–6% of tokens and gains 3.3 Exact Match points over raw documents on NQ (Less Is More). Ext2Gen trains the reader itself with Direct Preference Optimization to extract evidence from up to 25 noisy chunks before answering (Aligning Extraction and Generation). LDAR chooses a band of passages from retriever scores alone (Beyond RAG vs. Long-Context). Our approach differs from each. Unlike CORE-RAG, it uses no gold QA and no reader reward. Unlike Ext2Gen, it leaves the reader untouched and varies distractor structure on purpose, whereas Ext2Gen mixes benchmark chunks without such control. Unlike LDAR, it works at the sentence level and reads the text.
Research Program
The generator starts from real Wikipedia passages, so the surface text is organic. This addresses the concern that synthetic phrasing leaks shortcuts. For each passage, a QACG-style pipeline picks an evidence sentence, generates a question, and extracts the answer. Three kinds of distractor sentences are then added. The first are near misses, built by subject swap and relation swap through entity substitution and by pulling real sentences that share the question's relation but concern other entities. The second are topical filler sentences from the same and linked articles. The third are random sentences from unrelated text. Each training context randomizes the number of distractors, the mix of the three kinds, the position of the evidence, the chunk boundaries, and whether any evidence is present at all. Contexts with no evidence teach the filter to abstain.
The filter is a small sentence scorer, with the same backbone class as RECOMP's extractive compressor. It is trained with a contrastive loss that ranks evidence sentences above all distractors, plus a threshold for abstention. The reader is frozen and never appears in training.
A second-phase extension covers two-hop questions, built by chaining two generated questions through a bridge entity. We treat it as a test of a known weakness of synthetic generators (Zero-shot Fact Verification by Claim Generation), not as a promised win.
Baselines
The naive reference point is raw top-5 and top-20 retrieval passed straight to the reader, with no filtering. Every other method must beat this, and on NQ with semantic distractors it is the configuration at 37.81% in Rethinking Relevance.
The strongest existing method is CORE-RAG (Less Is More). It is the best competitor because it optimizes downstream accuracy directly and reports a gain over raw documents: +3.3 Exact Match on NQ. For HotpotQA, only a qualitative "better than full context" claim is available; no reported CORE-RAG HotpotQA EM number was found on Lacuna. We will retrain it with its published recipe on real NQ and HotpotQA training splits. It gets the same reader, the same retriever outputs, and a filter backbone of matched size. We give it at least as much training compute as our method, because RL is its strength. Beating it means matching its in-domain Exact Match within one point while losing less accuracy as distractors are added and when the reader family changes.
RECOMP's extractive compressor (RECOMP) is the closest architectural match. It is trained from reader feedback on real data, which makes it the right comparison for cross-family transfer. Its reported figures are about 36 tokens and under 10% relative EM loss on NQ. Ext2Gen (Aligning Extraction and Generation) is the strongest reader-modifying competitor, at 0.557 against a default of 0.473 with Llama3.1-8B-Instruct on its benchmark. We evaluate on that benchmark and report tokens processed, since Ext2Gen makes the reader read everything. LDAR (Beyond RAG vs. Long-Context) serves as the passage-level filtering baseline.
Three ablations isolate the key ingredient. The first trains the same filter on generated data with near misses removed, keeping only topical and random noise. The second keeps near misses but fixes density at one value. The third is the most important: the same filter trained on real hard negatives, meaning passages mined from the retriever's top results that lack the gold answer string. Training-example counts and steps are matched across all three.
Why not just…?
Why not just mine hard negatives from real retrieval instead of generating them? This is the real threat. Mined negatives are real near misses and need no generator. Their weakness is label noise. A top-ranked passage without the exact answer string may still support the answer through a paraphrase or an alias, and one that contains the string may not actually answer the question. Mined sets also cannot be pushed to controlled densities or reliably balanced between subject swaps and relation swaps. The third ablation settles it. If mined negatives match the synthetic filter on the distractor stress test and on cross-family transfer, we will report that the benefit comes from near-miss exposure and not from synthesis. In that case we would narrow the contribution to the curriculum design.
Why not just train CORE-RAG-style reinforcement learning on real data? It already works in-domain (Less Is More). Its reward, however, is the accuracy of one particular reader, which is exactly the kind of signal that failed to transfer across model families in RECOMP. The cross-reader experiment below tests whether that weakness also applies to CORE-RAG. If CORE-RAG transfers just as well, our advantage reduces to not needing labels, and we will say so.
Why not just feed everything to a long-context model, or use a bigger reader? The file's evidence argues against both. LDAR beats full 32K–128K contexts while using 25–63% of the tokens (Beyond RAG vs. Long-Context). Models at the 32B and 70B scale fall below 50% on CogniLoad once statement count exceeds 100 (CogniLoad). We will still run the unfiltered reader at every size we test, so readers can see directly whether a larger model makes the filter unnecessary.
Why not just keep random noise in the context, since it helped? The random-noise gain in Rethinking Relevance appeared in a low-accuracy setting, and it could reflect position effects rather than genuine contrast. We will test it directly: after filtering, we pad the context with random text and check whether accuracy rises. If it does, the output format should include padding. If it does not, that result is an informative negative finding.
Proposed Experiments
Evaluation covers NQ-open, TriviaQA, HotpotQA, and 2WikiMultihopQA with Contriever retrieval, along with the Ext2Gen benchmark and LongBench multi-document QA. We use three readers from different families: Llama 2 7B-Chat, Llama3.1-8B-Instruct, and a Qwen model. The metrics are Exact Match, tokens passed to the reader, abstention precision, and the slope of accuracy against the number of injected distractors.
The first decisive check replays the 18-semantic-distractor protocol of Rethinking Relevance. Success means recovering most of the gap between 37.81% and 56.42%, and doing so more than CORE-RAG and the mined-negative ablation.
The second check tests transfer across reader families. CORE-RAG and RECOMP filters are trained with one reader and evaluated with the other two. Our reader-free filter is evaluated with all three. The key quantity is the accuracy lost when the reader changes.
The third check compares the synthetic filter against the no-near-miss and mined-negative ablations. If near-miss synthesis adds nothing over mined negatives, the central claim fails.
The fourth check is the two-hop extension on HotpotQA. It tests whether chained synthetic questions help. A null result would confirm the limit already noted for synthetic generators.
References
- Rethinking Relevance: How Noise and Distractors Impact Retrieval-Augmented Generation
- RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation
- Domain Randomization for Scene-Specific Car Detection and Pose Estimation
- Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion
- When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density
- Efficient and Robust Question Answering from Minimal Context over Documents
- Zero-shot Fact Verification by Claim Generation
- Less Is More: Elevating RAG via Performance-Driven Context Compression
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding