Lineage-Conditional Claim Verification with Faithful Counterfactual Rationales

Many scientific facts are only true for a particular history. "This alloy has a Vickers hardness of 450" may hold after annealing and quenching but not after rolling alone. A fact checker that stores knowledge as timeless triples like (alloy, hardness, 450) cannot tell these cases apart. It will approve a claim that quotes a real measurement but attaches it to the wrong processing history. We propose to verify claims against a provenance graph, meaning a directed graph that records how each material or result was derived step by step. The verifier must return a label and also a rationale: the specific derivation steps in that graph that justify the label. A small domain ontology, a typed vocabulary of processing steps and the rules for how they can be ordered, grounds both the graph and the rationale.

The new part is not "use a graph." It is a testable faithfulness requirement on the rationale. If the verifier cites a processing step as the reason a claim is true, then replacing that step with a different but physically valid one should change the verdict. Editing a step the verifier did not cite should leave the verdict unchanged. We build a benchmark where this matters, train a verifier against that requirement, and test whether a frontier language model reading the same graph already meets it.

Core Claim

Claims whose truth depends on derivation history form a distinct verification problem, and it is invisible to static knowledge-graph checkers. The deciding test cases are "lineage-swapped" negatives: the value is real, but it is attached to the wrong derivation branch. We hypothesize two things. First, a verifier trained with counterfactual consistency over ontology-valid lineage edits will match strong prompted baselines on accuracy. Second, it will clearly exceed them on rationale necessity and invariance. The ontology constraint matters because, without it, counterfactual search drifts toward edits that flip the classifier but describe impossible histories. ProvX names this as its own open problem: perturbation-optimal subgraphs that do not match the real root cause.

What Prior Work Does and Leaves Open

Three lines of work each supply one piece. LitXBench supplies the data representation. It encodes alloy experiments as executable Python graphs of derivation steps, with validation checks such as "melting precedes casting." It reports that Gemini 3.1 Pro Preview extracts these at 0.80 overall F1. LitXBench stops at extraction, and we reuse its graphs and its semantic checks as the seed of our ontology.

ProvX supplies the counterfactual method. It learns edge masks on operating-system provenance graphs that flip a graph neural network's intrusion verdict, reaching 51.59% average necessity on DARPA TC. Its edits have no semantic typing, and its explanations are raw subgraphs of audit records. We keep the necessity metric but restrict edits to ontology-legal substitutions.

LKDA supplies the motivating negative result. In language-model and knowledge-graph fusion models, the graph is often a bystander: fidelity is 40.7% on CommonsenseQA before alignment. LKDA's Jensen-Shannon alignment raises fidelity to 98.0%. That alignment compares predictions with the graph branch attached and detached. Our objective instead compares predictions under targeted edits to the graph, which tests which evidence is used rather than whether the graph is used.

Two systems constrain the design, because they show that structure helps but leave history out. MedRAGChecker fuses text entailment with TransE plausibility scores over a static drug knowledge graph. When the graph signal overrode text entailment, its fused decisions agreed with experts 69.8% of the time versus 63.4% for text alone. Its triples carry no derivation context. EvoTrustRAG is the closest prior work. It builds a provenance-like graph of temporal precedence over text spans and classifies conflicts as evolution, intervention, or unresolved, reaching 79.1 macro-F1 on its diagnostic set. Its nodes are untyped strings and its provenance is temporal ordering. It does not ground steps in an ontology and does not test whether its attributions are causally necessary.

Research Program

Benchmark construction. We start from the gold LitXAlloy lineage graphs: 19 papers and 1,426 measurements. Each measurement is paired with its derivation path to make a supported claim. We then generate three kinds of negatives. Lineage-swapped negatives attach the same value to a sibling branch that exists in the same paper. Value-perturbed negatives change the number. Ontology-invalid distractors are histories that break the ordering rules. The lineage-swapped set is the core test.

The MPEA human-extracted data adds claims, but its values match LitXAlloy only about 87% of the time. We therefore treat MPEA as a noisy training source and use only LitXAlloy for evaluation. Nineteen papers is small, so all results are reported with paper-level cross-validation, so that no paper's graph appears in both training and test.

Verifier. The verifier is a mid-sized language model that reads the claim and a serialized lineage graph. It outputs a label (supported, refuted, or not enough information) and a pointer set over graph nodes and edges, which is the rationale. A natural-language explanation is produced from the pointer set, never independently. Every sentence of the explanation therefore maps to cited steps, which avoids the bystander pattern.

Counterfactual consistency training. For each training example, the ontology proposes valid edits. A cited step can be replaced with a same-type alternative that passes the LitXBench-style checks, or an uncited step can be edited. The ground-truth label of each edited graph is recomputed from the paper's real branches when possible. When it cannot be recomputed, the edited example is used only for the invariance term. Writing $p(\cdot \mid c, G)$ for the verifier's label distribution given claim $c$ and lineage graph $G$, the training objective adds two terms to the cross-entropy loss:

$$ \mathcal{L} = \mathcal{L}_{CE} + \lambda_1\, \mathbb{E}_{G' \in \mathcal{E}_{cited}}\, \mathcal{L}_{CE}(y', p(\cdot \mid c, G')) + \lambda_2\, \mathbb{E}_{G'' \in \mathcal{E}_{uncited}}\, \mathrm{JS}\big(p(\cdot\mid c,G)\,\|\,p(\cdot\mid c,G'')\big) $$

Here $\mathcal{E}_{cited}$ and $\mathcal{E}_{uncited}$ are the sets of ontology-valid edits to cited and uncited steps, $y'$ is the recomputed label of an edited graph, JS is Jensen-Shannon divergence, and $\lambda_1, \lambda_2$ are weights.

Transfer check. To test whether the method generalizes beyond materials, we apply it to the EvoTrustRAG diagnostic set. There, provenance is temporal ordering, and grounding means linking entity mentions to typed entities. This is secondary, and a null result there would narrow the claim rather than sink it.

Baselines

The naive reference is a value-match checker. It labels a claim supported if the claimed value appears anywhere in the paper's measurements, ignoring derivation history. By construction it scores at chance on lineage-swapped negatives. Its overall accuracy shows how much of the benchmark can be solved without lineage.

The static-knowledge baseline follows MedRAGChecker's design. It combines text consistency from AlignScore with a TransE plausibility score over flattened (material, property, value) triples, fused by the same calibrated logistic mixture. It is the strongest representative of the static paradigm. The only reported reference number is MedRAGChecker's 69.8% expert agreement on flipped decisions, which is in a different domain. We found no reported result on Lacuna for lineage-conditional claim verification.

The strongest competitor is a frontier language model given the full LitXBench-style code graph in its prompt and asked to reason step by step, then name the steps it relied on. LitXBench shows such models read these graphs well, at 0.80 extraction F1, so this baseline may well match us on accuracy. It receives the same graphs, the same few-shot exemplars, and a prompt-tuning budget matched to our hyperparameter search.

Our ablations isolate the two key ingredients. One trains the same verifier with cross-entropy only. Another uses unconstrained ProvX-style continuous edge masks instead of ontology-valid edits. A third replaces our edit-based term with LKDA's attached-versus-detached alignment.

Beating the strongest competitor means three things on the held-out lineage-swapped set. Accuracy must be within two points of the prompted model or better. Rationale necessity must be at least ten points higher. Invariance under uncited edits must also be higher. Necessity is the fraction of cases where a valid edit to a cited step flips the verdict, the same notion as ProvX's 51.59%.

Why not just…?

Why not just prompt a frontier model with the lineage graph and chain-of-thought? This is the real threat, and we will not argue it away. If the prompted model matches our verifier on both accuracy and the counterfactual metrics, the method contribution disappears and what remains is the benchmark plus a finding that prompting suffices. The evidence that it may fall short is indirect. LARK found that models override graph facts with what they learned in pretraining unless entity names are replaced by abstract symbols. In this domain, a model "knowing" that annealing usually softens alloys is exactly the kind of prior that could override a paper's specific branch. We test this directly. Each prompted-model rationale gets the same ontology-valid edits, and we measure whether its verdict follows the steps it cited. We also run a version with entity names replaced by symbols, to separate prior-override from simply ignoring the graph.

Why not just use a static knowledge graph verifier and add processing conditions as extra triple attributes? Attaching a "condition" field to triples is a simple patch, and we include it as a variant of the static baseline. Our expectation is that it handles single-step conditions but fails on multi-step paths where order matters. Annealing-then-rolling and rolling-then-annealing would share an attribute set, which is the connectivity loss Bresson et al. describe for flattened structure. The benchmark split by path length will show whether this patch is enough.

Why not just apply LKDA-style alignment, which already raised fidelity to 98%? LKDA measures whether the graph branch matters at all, not whether the specific cited evidence is the evidence used. A model can depend heavily on the graph and still cite the wrong step. Our ablation that swaps in LKDA's term answers this. If it matches our edit-based term on necessity, our objective adds nothing and we will report that.

Decisive Checks

Three outcomes decide the proposal. First, on lineage-swapped negatives, the static baseline must perform near chance while lineage-reading methods do not; otherwise the problem is not distinct. Second, our verifier must beat the prompted frontier model on necessity and invariance at comparable accuracy; otherwise prompting is sufficient. Third, the ontology-constrained ablation must produce counterfactuals that materials experts judge physically plausible, and that they agree identify the actual cause, more often than unconstrained masks do. This last check tests whether grounding the edits in an ontology addresses ProvX's root-cause disconnect. The experts rate a blinded sample of 200 counterfactual explanations from each variant.