Language models answer questions far worse when the prompt also contains plausible but irrelevant text. One reported case is a drop from 85.7% to 22.0% accuracy on CommonsenseQA for GPT-4o-mini under targeted distractions. The most reliable fix found so far is preference fine-tuning on pairs of clean and distracted examples. That leaves an open question: where should the training distractors come from? Current answers are expensive. One is an adversarial tree search that writes distractions aimed at the model's own weaknesses. Another is to retrieve the nearest neighbors of the gold passage from a vector database. A separate body of work finds that random selection of training data is very hard to beat, because clever selectors pile up odd, atypical examples and drift away from the real data distribution.
We propose a middle path and a test of whether it is needed. The idea is to sort candidate distractors by how they relate to the gold evidence in a concept hierarchy: siblings, more general ancestors, more specific descendants, distant cousins, and unrelated text. A hyperbolic embedding makes this sorting possible. Hyperbolic embeddings are vector spaces with negative curvature, where distance from the origin tracks how specific a concept is and distances expand exponentially near the boundary, so tree-like data fits without crowding. Within each group we then sample uniformly at random. Our claim is that robustness to distraction depends mainly on covering the right hierarchical relations, not on picking the hardest individual distractors. If that is true, uniform sampling within hyperbolic strata should match adversarial search at a small fraction of the cost and should avoid the loss of general capability that aggressive selection causes. The design includes the ablation that could make the geometry unnecessary.
Core Claim
Two findings in the literature pull against each other, and the question sits between them. Understanding Hyperbolic Metric Learning through Hard Negative Sampling shows that hyperbolic geometry's advantage in contrastive image retrieval comes from acting as a built-in hard-negative miner. Negatives near the boundary exert exponentially larger gradients. Random Selection Reveals Implicit Knowledge Consensus in Code Generation and Random Is Hard to Beat: Active Selection in Online DPO with Modern LLMs show the opposite pattern for training data:
- Diversity-maximizing and uncertainty-based selection fall into a "diversity trap" of atypical boundary cases.
- Active preference selection costs 20.2x more compute for negligible win-rate gain.
- Aggressive selection can hide a capability collapse, with accuracy dropping more than 10% on general benchmarks while proxy win-rates exceed 0.9.
Our hypothesis reconciles the two. Hardness is useful as a way to organize the pool, harmful as a way to pick examples. Concretely, we predict three orderings:
- Reader robustness will be highest when training distractors are drawn uniformly within hierarchy-relation strata, with the strata defined in hyperbolic space.
- Taking only the hyperbolically hardest negatives in each stratum will overfit, reproducing the diversity trap.
- Plain uniform random distractors will be too easy to teach the model to reject siblings and parents, which HypRAG identifies as the documents that actually confuse readers.
Prior work plays three roles here:
- Motivation. HypRAG and HyperbolicRAG establish that hierarchy-related distractors matter. Both work only on the retriever side and keep the answering model frozen.
- The training recipe. Breaking Focus and Aligning Extraction and Generation show that preference fine-tuning on clean versus distracted examples works. Neither compares its distractor source to uniform sampling.
- The constraint. The random-baseline literature warns that any curation step must beat uniform sampling at equal compute, or it is not worth using.
Research Program
Building the hierarchy map. We embed a Wikipedia paragraph pool and the MeSH-structured biomedical corpus with a hybrid hyperbolic encoder, following HypRAG's HyTE-H design: a Lorentz projection on top of a Euclidean encoder, pooled with the Outward Einstein Midpoint so that specific passages are not pulled toward the origin. We fine-tune it with a parent–child containment loss of the kind HyperbolicRAG uses. For a gold passage $g$ and candidate $x$, we compute two quantities. The first is the geodesic distance $d(g,x)$. The second is the radial offset $\Delta r = \|x\|-\|g\|$, the difference in distance from the origin. Candidates are assigned to strata as follows:
- Siblings: small $d$ and $\Delta r \approx 0$.
- Ancestors: small $d$ and $\Delta r<0$, meaning the candidate is more general.
- Descendants: small $d$ and $\Delta r>0$.
- Cousins: moderate $d$.
- Unrelated: large $d$.
We validate the strata against the true MeSH tree before using them anywhere else.
Building training data. Questions come from the HotpotQA and MuSiQue training splits with their gold paragraphs. Each training context mixes the gold evidence with $k$ distractors. Several variables are held fixed across every condition that changes the distractor source:
- the number of preference pairs;
- the value of $k$;
- the proportions across strata;
- the DPO steps and hyperparameters.
DPO (direct preference optimization) trains a model directly on chosen-versus-rejected answer pairs. Chosen responses give the correct answer and cite the gold sentence, in the style of Ext2Gen. Rejected responses are the model's own on-policy answers under distraction.
Analysis. We build a train-stratum × test-stratum transfer matrix, which shows which relation types generalize to which. We also do a mechanistic check in the spirit of Llama See, Llama Do. It measures how attention mass on gold versus distractor tokens shifts after each training condition.
Baselines
Naive reference points. The first is the untuned Llama-3.1-8B-Instruct reader. The second is the same reader with an explicit "ignore irrelevant information" instruction. Breaking Focus reports that this instruction gives almost no protection, so it serves as the floor.
Uniform random distractors. Distractors are drawn from the whole corpus with no structure. This is the baseline the random-selection literature says is hard to beat, and it costs almost nothing. Beating it requires a statistically significant gain on the sibling and ancestor test strata and on the adversarial test set, with no loss on clean accuracy.
Euclidean top-k hard negatives. This follows Ext2Gen's recipe of vector-database retrieval. The file reports Llama3.1-8b-instruct rising from an average score of 0.473 to 0.557 on the Ext2Gen evaluation. We rerun this recipe on our pairs, with the same pair count and the same DPO budget.
Euclidean-stratified random: the key ablation. Strata are built from bands of Euclidean cosine-similarity rank, similar to the rank bands in LDAR, and we sample uniformly within them. This condition separates two ingredients: stratifying the pool at all, and stratifying it with hyperbolic geometry. The hyperbolic version must beat it specifically on the sibling and ancestor strata to justify the geometry.
Hyperbolic hardest-negative. This uses the same strata but takes the closest candidates in each. It tests whether the geometry's built-in hard mining falls into the diversity trap.
Adversarial tree search. This is the strongest existing method, from Breaking Focus and Adaptive Distraction. The file reports that DPO training on its distractions improves accuracy on perturbed inputs by "nearly 50% in some smaller models." We regenerate its distractors with the published procedure. It receives the same number of pairs, and we report its generation compute separately. Our method counts as better if it matches this baseline's robustness within confidence intervals at lower data-generation cost, or exceeds it on transfer to retrieval noise.
Why not just…?
Why not just use uniform random distractors? This is the most serious threat, given the code-generation and online-DPO results. The counterargument is that distraction data differs from those settings. In multi-hop QA, dangerous distractors are rare in a random draw from Wikipedia. MuSiQue shows that models taking single-hop shortcuts lose about 30 F1 points, which suggests that near-miss passages drive the errors. Still, if random distractors match our method on the sibling test stratum, we will report that outcome as the main result.
Why not just lower the temperature or bin by Euclidean similarity? The metric-learning paper reports two relevant findings. Euclidean models match hyperbolic ones only at a finely tuned temperature near 0.05. The two geometries also surface different sets of hard negatives. HypRAG reports 96.3% versus 81.0% for hierarchy-sensitive target identification on MeSH. None of that evidence concerns training an answering model, so the Euclidean-stratified ablation decides the question directly.
Why not just fix retrieval at test time with HypRAG or Superposition Prompting? The two approaches are complementary but address different failures. Test-time filtering cannot help when the distractor is inside the prompt by construction, as with the tree-searched sentences. Superposition Prompting also isolates documents from one another, which fits poorly with multi-hop bridging. We evaluate our trained reader both with and without a hyperbolic retriever to measure whether the gains add up.
Why not just use tree-search distractors? They target a specific model and are expensive to generate. The random-selection literature predicts they will overfit and erode general capability. We measure that erosion directly, using LM Eval Harness normalized accuracy, the metric Random Is Hard to Beat uses for its capability-collapse finding.
Proposed Experiments
Test suites. We evaluate on four:
- HotpotQA distractor setting. The original BiDAF baseline reaches answer F1 58.28 and humans reach 91.40.
- MuSiQue-Ans. HyperbolicRAG reports EM 39.5 and F1 50.6 with a frozen reader.
- The MeSH hierarchy benchmark, with injected distractors separated by stratum.
- Tree-searched distractions on CommonsenseQA and MMLU, as an out-of-distribution adversarial transfer set.
We also track capability retention on MMLU without distractors.
Decisive checks. Three checks determine the outcome. First, hyperbolic-stratified random must beat Euclidean-stratified random on the sibling and ancestor strata. If it does not, we conclude that stratification alone is enough. Second, hyperbolic-stratified random must beat hyperbolic hardest-negative on clean accuracy and capability retention. If it does not, the diversity-trap reconciliation fails. Third, the method must reach within confidence intervals of tree-search robustness on the adversarial transfer set at lower generation cost.
All conditions use three seeds and report bootstrap confidence intervals. The transfer matrix will show whether training on siblings alone accounts for most of the benefit. If it does, that would be an even simpler recipe than ours.
References
- Understanding Hyperbolic Metric Learning through Hard Negative Sampling
- Random Selection Reveals Implicit Knowledge Consensus in Code Generation
- Random Is Hard to Beat: Active Selection in Online DPO with Modern LLMs
- HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation
- HyperbolicRAG: Enhancing Retrieval-Augmented Generation with Hyperbolic Representations
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
- Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation