Large language models are increasingly used to pull structured facts out of text, in the form of triples like (entity, relation, entity), for example (Aspirin, interacts-with, Warfarin). They make these triples up at alarming rates. When a sentence mentions two scientific terms that are not actually related, a leading LLM still asserts a relation between them 96.9% of the time on the SciERC benchmark (DEPTH). This proposal asks a diagnostic question that existing fixes skip. When the model outputs a relation, was it read from the sentence, or remembered from what the model already believes about those two entities? It may also have been attached to the wrong pair of entities. We propose to answer this for every extracted triple by looking inside the model. We would measure how much the relation decision depends on the evidence words in the text, and how closely it matches the relation the model produces for the same entity pair when no text is given.
The bet is that these three failure types differ, and that the difference is visible in the model's internal states. Interpretability work shows that LLMs build a relation representation in middle layers that does not depend on the subject (Locating and Extracting Relational Concepts). Work on retrieval-augmented generation shows that hallucinations often come from feed-forward layers injecting stored knowledge over correctly attended context (ReDeEP). If relation extraction behaves the same way, a per-triple "read versus remembered" score would do three things. It would detect hallucinated triples without labeled data. It would tell us which failure type dominates each benchmark. It would also suggest a targeted fix: remove the remembered relation rather than suppress relations across the board.
Core Claim
The claim has three parts. First, most LLM relation hallucinations on pairs with no real relation are driven by the model's stored belief about the entity pair, not by misreading the sentence. Second, a training-free score can separate grounded triples from these hallucinations better than the model's own confidence. The score combines two things: the causal dependence of the relation decision on the evidence tokens, and the alignment of the middle-layer decision state with a context-free "prior" state for the same pair. Third, the remaining hallucinations are mis-binding errors, which have a different signature. In a mis-binding error, the model reads a real relation from the sentence but attaches it to the wrong entities, much like the donor/patient swaps observed in clinical generation (MEDIMP). The signature is strong dependence on text, but on a span belonging to other entity mentions.
What would be new is threefold. The first is a per-triple causal decomposition of extraction decisions into read, remembered and mis-bound. The second is a knowledge-conflict relation-extraction test set that makes these sources separable. The third is an honest test of whether looking inside the model buys anything over simply rerunning it on edited inputs.
Relation to Prior Work
Two works motivate the gap. DEPTH documents the no-relation failure and cuts the average hallucination rate across eight benchmarks from 18.4% or higher to 7.9%. It does this by simplifying sentences to the dependency path between the entities and by RLHF (reinforcement learning from human feedback) training with a causality-driven reward model. It never asks why the model hallucinates, needs an external parser, and requires expensive training. We use its benchmarks and its hallucination-rate metric, but work on hidden states and require no fine-tuning for detection. ReDeEP separates context contribution from stored-knowledge contribution for free-form retrieval answers, reaching 0.7458 AUC on RAGTruth. Its scores are pooled over a whole generated chunk and know nothing about entity pairs. We condition the decomposition on a specific pair and a specific evidence span. We also add a third category, mis-binding, that its two-way split cannot express.
Three works supply the method. The subject-agnostic relation representation and the hidden-state transplantation result from Locating and Extracting Relational Concepts tell us where to look. Transplanted relation states redirect predictions with 80–90% accuracy in layers 9–17 of Llama2-7b and Mistral-7b. Identifying Linear Relational Concepts shows that relation concepts live in a low-rank subspace, with performance peaking at rank around 200. We use this to project decision states before comparing them, so that the prior-alignment score is not dominated by unrelated activation noise. Path patching and the activation patching best-practices study give us the intervention machinery and metric choices.
One work constrains the design. Path patching only establishes sufficiency on a chosen distribution, not completeness, so redundant circuits could hide context dependence. We therefore validate our scores against behavioral ground truth on constructed data rather than trusting the circuit story. ROME also constrains us. Its finding that relations are stored one-directionally means the prior state must be computed with the entity pair in the same order as the extraction query.
Three probing methods target hallucination on question answering but never touch extraction. CLAP is a supervised cross-layer probe at the end-of-sentence token. Truth Forest uses orthogonal truth directions. TruthFlow learns query-specific corrections. TruthFlow's observation that corrections point in query-specific directions supports our choice of a pair-specific prior rather than a global "hallucination direction."
Research Program
The extractor is an open 7B model, Llama2-7b and Mistral-7b, matching the models where relation representations have been mapped. It is prompted with a sentence, two marked entity mentions, and the relation label set, and it emits a label including NO-RELATION. The decision position is the token where the label is generated.
For each triple we compute two quantities. The first is context reliance. We knock out attention from the decision position to the evidence tokens, meaning the tokens on and around the path between the two mentions. We then measure the drop in the predicted relation's log-probability, using the patching metrics recommended in the best-practices study. We repeat the knockout on spans attached to other entity mentions to obtain a mis-binding reliance.
The second is prior alignment. We run the same prompt with the sentence replaced by length-matched neutral filler, which gives the model's remembered relation for that ordered pair. We then take the cosine between the middle-layer decision states of the two runs, after projecting both onto the low-rank relation subspace. The layer band is chosen by sweeping layers 4–17 on a development split, guided by the transplantation results.
A triple with low context reliance and high prior alignment is labeled remembered. A triple whose reliance falls mostly on another mention's span is labeled mis-bound. A triple relying on its own evidence span is labeled grounded. For a continuous detector, we combine the three numbers with a two-parameter logistic fit on a small development set. This follows the ReDeEP recipe, so no hidden-state probe is trained.
The mitigation arm tests whether diagnosis earns its keep. For triples flagged as remembered, we subtract the projection of the prior state from the decision state at the chosen layer, in the spirit of the relational steering in the LRC work, and let the model re-decide. For triples flagged as mis-bound, we mask attention to the competing mention's span.
To make the sources separable, we construct a knowledge-conflict test set from the DEPTH eight-benchmark suite: TACRED, TACREV, Re-TACRED, NYT11, SciERC, FOBIE, DDI and SemEval. It has three kinds of items. The first are no-relation sentences where we substitute entity pairs that have a strong real-world relation. We confirm that the model states this relation in the no-context run, so a hallucination there must be remembered. The second are sentences asserting a relation between pairs with no stored association, so any correct prediction must be read. The third are multi-entity sentences where we swap mention order to create controlled mis-binding opportunities. The constructed labels give ground truth for the decomposition itself, independent of our scores.
Baselines
The naive reference point is plain zero-shot prompting of the same backbone, with the model's NO-RELATION probability used as a detection score. DEPTH reports that a leading LLM hallucinates on 96.9% of SciERC no-relation pairs under this regime, and we expect a comparably poor starting point. Beating it means a lower hallucination rate at equal or higher micro-F1.
The strongest existing method on this exact task is DEPTH. It reports a 7.1% hallucination rate on SciERC, 7.9% averaged across the eight benchmarks, and 12.9% with a 3B backbone. Its reported numbers come from its own backbones and RLHF training, so we will make the comparison fair in two ways. We cite the reported figures directly. We also reimplement its training-free component, shortest-dependency-path simplification plus global refinement prompting, on our 7B backbones with the same data and prompt budget. Our training-free detection-plus-mitigation pipeline counts as competitive if it matches the reimplemented DEPTH on hallucination rate within two points without any RLHF. It counts as clearly better if it also closes most of the gap to DEPTH's reported 7.9% average.
The strongest interpretability competitor is ReDeEP applied unchanged, with chunk-level context and stored-knowledge scores computed over the emitted label. It achieves 0.7458 AUC on RAGTruth for LLaMA2-7B, and we found no reported result on Lacuna for ReDeEP on relation extraction. This baseline also serves as the key ablation. It is our method minus pair-conditioning and minus the prior-state comparison, so the gap between them isolates the new ingredient.
A supervised CLAP-style cross-layer probe is trained on the same development labels we use for the logistic fit. It serves as an upper-bound-style competitor that uses labels far more heavily. No reported result was found on Lacuna for CLAP on relation extraction.
All detectors are scored by AUROC for separating hallucinated from correct triples, overall and separately on the remembered, read and mis-bound slices of the conflict set. All get the same development split and the same hyperparameter search budget.
Why not just…?
Why not just rerun the model without the sentence and flag triples it still predicts? This black-box context-removal test is the most serious threat. It directly estimates the remembered relation, and for the first kind of conflict item it may match our prior-alignment score. We include it as a baseline. If it equals our detector on remembered hallucinations, we will say the mechanistic prior score adds nothing there. The expected shortfall is elsewhere. Context removal cannot distinguish a grounded prediction that happens to agree with memory from a remembered one, because both survive removal. It also says nothing about mis-binding. The per-slice AUROCs settle this.
Why not just occlude text spans and rerun, instead of knocking out attention? Deleting words is the black-box analog of our context-reliance score and needs no model access. Its weakness is that deletion produces ungrammatical inputs that shift the whole computation, which is the syntactic-noise problem DEPTH attributes hallucinations to in the first place. We run occlusion with the same span definitions. If it matches attention knockout on the constructed ground truth, we will report the mechanistic version as unnecessary for this component.
Why not just train a probe on labeled hallucinations, CLAP-style, or fine-tune like DEPTH? Supervised probes may well detect better in-distribution. CLAP is strong precisely because it learns from labels. But a probe outputs a single score and cannot say whether an error was remembered or mis-bound. It also needs labels per domain, and DEPTH's RLHF is costly. Our claims are about diagnosis and label-free transfer, so the decisive test is cross-benchmark transfer. We fit on TACRED and evaluate on SciERC, DDI and FOBIE, where the entities and priors differ.
Proposed Experiments
The first experiment validates the decomposition. On the constructed conflict set, where the source of each hallucination is known by construction, we measure how accurately the read/remembered/mis-bound labels recover the true source. We compare against the black-box context-removal and occlusion baselines.
The second is the diagnostic study. Across the eight benchmarks, we report what fraction of real no-relation hallucinations fall in each category. The prediction is that remembered errors dominate on SciERC and DDI, where technical priors like "method used-for task" are strong. It is falsified if mis-binding dominates. In that case we would conclude that DEPTH's syntactic diagnosis is the right one, and the stored-knowledge story from ReDeEP does not carry over to extraction.
The third experiment is detection, measured as AUROC against all baselines above, including cross-domain transfer. The fourth is mitigation: hallucination rate and micro-F1 after targeted prior subtraction and binding masks. We compare against uniform suppression of relation predictions to check that gains are not bought by saying NO-RELATION more often. This is the extraction analog of the informativeness-versus-truthfulness trade-off noted for TruthFlow.
The decisive checks are two. First, the pair-conditioned score must beat unmodified ReDeEP by a clear AUROC margin. Second, it must beat context removal on the mis-bound and read-but-agreeing-with-memory slices. If both fail, the honest conclusion is that relation hallucination is a black-box-diagnosable problem, and the proposal collapses to the cheaper tool.
References
- Hallucination-Resistant Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement
- Locating and Extracting Relational Concepts in Large Language Models
- ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability
- MEDIMP: 3D Medical Images and clinical Prompts for renal transplant representation learning
- Identifying Linear Relational Concepts in Large Language Models
- Localizing model behavior with path patching
- Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
- Locating and Editing Factual Associations in GPT
- Cross-Layer Attention Probing for Fine-Grained Hallucination Detection
- Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuning
- TruthFlow: Truthful LLM Generation via Representation Flow Correction