Language models often get the facts in a passage right but assign them to the wrong participant. The drug that was given becomes the drug that interacted. The donor's condition is attributed to the recipient. The method that was compared becomes the method that was proposed. We propose that many of these "role swaps" are not free-floating factual errors. Instead, they come from a failure that psycholinguists have studied for decades: an attractor, meaning an intervening word or phrase that looks like a plausible filler for the role, sits between a predicate and its true argument and captures the link. We will test this with controlled edits to real sentences. Holding linear distance fixed, we vary only whether the intervening material contains an entity of the same type as the true argument. We will then check whether the model's errors land on that specific entity.
If the account holds, it points to a cheap mitigation that needs no parser. Replace only the same-type intervening entities with neutral typed placeholders before asking the question, and keep everything else. The decisive comparison is against pruning the sentence to its dependency path, the key ingredient of the strongest existing method. If placeholder masking recovers most of that gain while deleting far fewer tokens, and random deletion of the same number of tokens does not, then attractors rather than length are what drive this class of hallucination.
Core Claim
We claim three things. First, in context-grounded extraction and question answering, role-swap errors in large language models (LLMs) rise with the number of feature-matched attractors between a predicate and its argument, and much less with linear distance filled by non-competing material. Second, when the model errs, the wrong argument it produces is disproportionately the attractor, and more so when the attractor matches the true argument in entity type and grammatical number. Third, an intervention that neutralizes only the attractors captures most of the benefit of full structural simplification, without depending on a dependency parser that fails on technical text. The first two claims are diagnostic and can be falsified independently of the third.
How This Relates to Prior Work
Two lines of work motivate the hypothesis. On the syntax side, Exploring the Syntactic Abilities of RNNs with Multi-task Learning showed that recurrent networks' subject-verb agreement collapses to 50.6% in relative clauses with intervening attractors, and recovers to 76.2% with auxiliary grammar supervision. The importance of Being Recurrent for Modeling Hierarchical Structure found that pure attention models are more, not less, vulnerable to attractors as distance grows. RuBLiMP reports that modern LLMs still degrade on natural minimal pairs involving agreement across attractors and on distant pronoun links. All three measure grammatical agreement, not factual content. We carry their controlled-attractor design into the regime where the error is a false statement about who did what.
On the hallucination side, Factual Inaccuracy and Hallucination in Large Language Models lists entity and role swapping as a major hallucination type but treats it descriptively. Sources of Hallucination by Large Language Models on Inference Tasks traces hallucinated entailment to two pre-training shortcuts: memorized sentences keyed by named entities, and relative predicate frequency. It explicitly leaves syntactic distance unexamined. We add structural interference as a third, testable source, and we check whether it interacts with that paper's memorization effect.
The closest work, and the one that constrains our design, is DEPTH. DEPTH prunes each sentence to the shortest dependency path between a candidate entity pair and trains the LLM on these pruned inputs with a causality-driven reward model. It cuts average relation-extraction hallucination to 7.9% and SciERC no-relation hallucination from 96.9% to 7.1%. DEPTH shows that removing off-path material helps. It does not say which removed material matters, and it depends on an upstream parser. Its target is also the "null bias," where the model invents a relation when none exists, rather than attachment to the wrong participant. We decompose DEPTH's input-side ingredient into "remove attractors" versus "remove everything else," and we do so without a parser.
GPT-RE supplies the strongest prompting-only method. It retrieves demonstrations using a fine-tuned relation model's entity-aware embeddings and adds gold-label reasoning to them, but it leaves the query sentence's own distractors untouched. Learning Latent Forests for Medical Relation Extraction motivates avoiding fixed parsers: learned structure beats parser trees most clearly on sentences over 50 words. It does so with an expensive differentiable tree computation that we deliberately avoid.
Research Program
The first component is a controlled corpus of edits. We start from test sentences in Re-TACRED, SciERC, and DDI (drug-drug interaction), each with a gold relation and argument pair. For each sentence we create matched variants at a fixed insertion point between predicate and argument:
- an attractor insertion, which adds an appositive or relative clause containing an entity of the same type as the true argument;
- a mismatched insertion, identical except that the inserted entity has a different type or grammatical number;
- a neutral padding, which adds a clause of equal token length with no entity.
We produce variants with zero, one, and two attractors. An LLM generates the inserted clauses. Human annotators then verify on a sample that the gold relation is unchanged and that the sentence stays natural. We filter for low-likelihood phrasing using the MIN-K% probability check that RuBLiMP used against memorization.
Using Re-TACRED rather than TACRED matters here. Re-TACRED documents a 23.9% label error rate in the original, which would swamp role-level effects.
The second component is the measurement. We query each variant in two formats. The extraction format asks for the relation and its arguments. The question-answering format asks, for example, "Which drug interacts with X?" We score each response in three ways:
- role-swap rate: the relation type is correct but an argument is wrong or reversed;
- attractor-capture rate: among wrong arguments, the fraction that are the inserted attractor;
- standard micro-F1.
We run open-weight models from the same family at several sizes, so that scale can be separated from structure. As a secondary analysis, we make the attractor a well-known entity that the model has likely seen often, to test whether interference amplifies the memorization-driven bias described in Sources of Hallucination.
The third component is an observational check on organic errors. We annotate the role-swap hallucinations in FaithBench, whose summaries come from ten modern LLMs. For each swapped fact, we count feature-matched entities in the source between the relevant predicate and its true argument. We then fit a regression of swap occurrence on attractor count, linear distance, and sentence length. This tests whether the controlled effect appears in errors nobody engineered.
The fourth component is the intervention. Attractor masking runs an off-the-shelf named-entity tagger. It replaces every same-type entity between the queried predicate and the queried arguments with a typed placeholder such as "[OTHER-DRUG]", and it keeps a mapping so that answers can be restored to the original names. No dependency parse is used; the tagger only needs the queried pair's positions.
We also build a risk score, the attractor count on the argument-predicate span. It serves as a feature that flags claims for verification. We will test whether adding it to an existing detector improves detection on FaithBench's hard cases.
Baselines
The naive reference is the same LLM, zero-shot, on the unmodified sentence. On the controlled corpus it fixes how much each insertion type hurts. The nearest reported point for plain prompting is GPT-3 with standard sentence-level demonstration retrieval, which scored about 33.45 micro-F1 on TACRED in GPT-RE.
The strongest existing input-side method is dependency-path pruning in the style of DEPTH. We parse each sentence and keep only the shortest dependency path between the entity pair plus the entities themselves. We then feed the result to the same model, with the same prompt and no additional training.
DEPTH's reported numbers include reward-model and PPO training that we do not replicate: 7.1% no-relation hallucination on SciERC, 7.9% average hallucination, and a 9.3-point average F1 gain. Our comparison therefore isolates its pruning ingredient under identical inference conditions. It is not a claim to beat the full trained system. Beating this baseline means matching its role-swap reduction within confidence intervals while deleting substantially fewer tokens. It also means outperforming it on the subset of SciERC and DDI sentences where the parser's path is visibly wrong, identified by comparing against gold arguments.
The strongest prompting method is GPT-RE's task-aware retrieval with reasoning-augmented demonstrations. It reports 69.00 micro-F1 on SciERC and 72.14 on TACRED. We run it with the same backbone and the same demonstration pool, then run it again with attractor masking applied to the query. That tests whether the two effects add up. Beating it means that masking plus GPT-RE improves role-swap rate over GPT-RE alone on attractor-bearing sentences.
The key ablation is length-matched random deletion. It removes the same number of intervening tokens as attractor masking, chosen at random among non-entity words. A second ablation, mismatched-entity masking, masks intervening entities of the wrong type. If either recovers as much as attractor masking, the mechanism claim fails.
For detection, the baseline is the best detector on FaithBench's hard cases, RAGAS with GPT-4o at 62.31% balanced accuracy. We will test whether adding the attractor-risk feature raises that figure on the role-swap subset.
All methods get the same backbone, decoding settings, and prompt-tuning budget, with prompts selected on a shared development split.
Why not just…?
Why not just scale the model? RuBLiMP reports that model performance drops by up to 12% once memorized material is filtered out, and that some size increases bring little agreement gain. On the hallucination side, HalluLens (HalluLens) shows that even Llama-3.1-405B trades hallucination for a 56.77% false refusal rate on valid questions. Neither result settles role swaps. Our multi-size runs answer the question directly. If the attractor effect shrinks to nothing at the largest size, we will report that structure matters only for smaller models, which narrows the claim rather than hiding it.
Why not just use chain-of-thought prompting? FaithBench found that chain-of-thought judges often underperform plain zero-shot prompts because the reasoning introduces new hallucinations. A reasoning chain that restates "the drug that interacts with X is…" can itself be captured by the attractor. We include chain-of-thought as a baseline on the controlled corpus. The informative measurement is whether its errors still land on the attractor.
Why not just ask the LLM to rewrite the sentence into simple clauses first? This is the most serious threat. It is parser-free, general, and plausibly effective, and the file contains no reported result for it. We run it as a baseline. If LLM self-simplification matches attractor masking, the mitigation contribution collapses, and what remains is the diagnosis plus a cheaper, more transparent rewrite. We will also check whether the rewrite step itself produces role swaps on high-attractor sentences. The interference hypothesis predicts that it will, which would favor masking done before any generation.
Why not just prune to the dependency path, as DEPTH does? Pruning plausibly works, and it is our main baseline. The question is whether it works because it removes attractors. Pruning also inherits parser errors, a limitation both the DEPTH analysis and the Learning Latent Forests for Medical Relation Extraction results highlight for technical text. The random-deletion ablation and the parser-failure subset separate these two explanations.
Why not just use token-level uncertainty to flag risky answers? Model confidence may already rise on attractor-heavy inputs, which would make a structural score redundant. No reported result on Lacuna covers uncertainty-based role-swap detection. We will compare the attractor-risk feature against the answer's token probability. We report whether structure adds predictive value once confidence is controlled for.
Decisive Checks
The account is supported only if four things hold on the controlled corpus. Attractor insertion must raise role-swap rate clearly more than length-matched neutral padding. Attractor-capture rate among errors must be well above what random choice among the sentence's same-type entities would give. Mismatched-type attractors must produce a weaker effect than matched ones. And attractor masking must recover most of the dependency-path pruning gain while random deletion of the same length does not.
The account is falsified in two cases. If neutral padding hurts as much as attractor insertion, the problem is distance, not interference. If errors do not concentrate on the attractor, the problem is general confusion, not capture. On FaithBench, the attractor count should predict organic role swaps beyond sentence length and distance, and it should add measurable balanced accuracy over the 62.31% detector on that subset.
References
- Exploring the Syntactic Abilities of RNNs with Multi-task Learning
- The importance of Being Recurrent for Modeling Hierarchical Structure
- RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
- Sources of Hallucination by Large Language Models on Inference Tasks
- Hallucination-Resistant Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement
- GPT-RE: In-context Learning for Relation Extraction using Large Language Models
- Learning Latent Forests for Medical Relation Extraction
- Re-TACRED: Addressing Shortcomings of the TACRED Dataset
- FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
- HalluLens: LLM Hallucination Benchmark