Distilling Claim Verification into Dependency Trees That Show Their Work

A fact-checking system should say which part of a claim fails, and that explanation should be the actual computation that produced the verdict, not a heat map added afterward. This note proposes a small verifier that is wired along the claim's grammar. It uses a dependency tree, a parse in which each word points to the word it modifies, so "not" attaches to its verb and a relative clause attaches to its noun. Every node in the tree carries a three-way truth state with respect to the evidence: supported, refuted, or not enough information. States are combined bottom-up, and the rule for each combination depends on the grammatical link. A negation link can flip a child's state. A conditional or quantifier link changes how children combine. The verdict is simply the state at the root, so the annotated tree is the explanation.

The obstacle is training signal. Datasets give one label per claim, and no one labels subtrees. We get dense supervision through knowledge distillation, in which a large "teacher" model supervises a small "student." We turn each subtree into a short sub-claim and ask a large instruction-tuned model whether the evidence supports it. Those soft judgments then train the student's internal nodes. The bet is that the teacher supplies broad lexical and world knowledge, and the tree student supplies the directional logic that large models handle poorly. Put in two sentences: a big model grades pieces of the claim, and a small grammar-shaped network learns to combine those grades correctly. The result should be faster, more robust on negation-heavy and conditional claims, and faithful by construction.

Core Claim

Two properties can be had together. Phrase-level soft-logic distillation, as in LOREN, gives faithful verdicts. Recursive composition along dependency arcs, as in AttentiveTreeNet, handles negation and monotonicity. Getting both requires dense teacher supervision on the internal nodes of the tree. Without it, a tree student trained only on claim-level labels learns shortcuts in its internal states, just as flat models do.

The claim is testable. On a subset of FEVER claims containing negation, conditionals, quantifiers, or long-distance modifiers, the distilled tree student should beat LOREN's flat AND/OR aggregation at equal or lower model size. Its node states should also localize the refuting phrase at least as well as LOREN does.

Where This Sits

Three strands of prior work motivate the gap.

  • Dependency structure helps logic but has not been turned into a fact-checker. Monotonicity Reasoning in the Age of Neural Foundation Models runs a Tree-LSTM over dependency parses and reaches 75.7% on MED, against 71.6% for BERT-base. It also reports GPT-3-class models at 50–58% on CURRICULUM. That paper trains from scratch on sentence pairs, with no distillation and no retrieval.
  • LOREN supplies the method we extend. LOREN assigns latent truth states to constituency chunks and distills soft AND/OR logic into a student. It reports a 72.93 FEVER score, over 96% agreement between verdict and phrase states, and 74.3 F1 at finding the culprit phrase. Its authors name dependency arcs, double negatives, and conditionals as unsolved. We replace flat chunks and fixed gates with recursive, arc-typed composition, and we replace purely self-consistent latent training with external teacher grades on subtrees.
  • Other structured verifiers keep a flat or hand-built structure. QA-NatVer aligns flat chunks and prompts Flan-T5 for natural-logic operators at inference time. We use a similar large model only offline, as a teacher. Fact Checking Machine Generated Text with Dependency Trees (DepChecker) is the only existing dependency-based fact-checker. It is purely rule-based and fast, about 72 ms per sentence, and its authors report it breaks on lexical variation. We keep its dependency-path traces but learn the rules.

Two papers constrain the design.

  • Structural distillation usually throws the structure away. Mimic and Conquer distills tree teachers into a flat LSTM and finds the flat student matches or beats the trees while being robust to parse noise. That is a direct warning that an explicit tree at test time may cost accuracy. We accept the risk because that student can no longer output a structural justification, and we measure the cost directly.
  • Soft teacher labels may carry little structure. Knowledge Distillation ≈ Label Smoothing questions whether soft labels carry structure at all. This is why we distill node-level judgments rather than just claim-level logits, and why we include a label-smoothing control.

Research Program

Parsing and the student network. Each claim is parsed with an off-the-shelf dependency parser. The student encodes claim and retrieved evidence tokens with a small pretrained encoder. We use evidence from LOREN's released retrieval pipeline, so retrieval is held fixed across all systems.

Each tree node $v$ forms a vector $h_v$. It combines the node's own token representation, cross-attention from that token to the evidence, and its children's vectors. Each child $c$ passes through a transform $W_{r(c)}$ indexed by its arc label $r(c)$, for example negation, clausal complement, conditional marker, or adjectival modifier. A head then reads the node's truth distribution:

$$q_v = \mathrm{softmax}\big(U\, h_v\big), \qquad h_v = f\Big(x_v,\ \mathrm{attn}(x_v, E),\ \textstyle\sum_{c} W_{r(c)} h_c\Big),$$

Here $x_v$ is the token embedding, $E$ is the evidence encoding, $f$ is a Tree-LSTM cell, and $q_v$ is a distribution over {supported, refuted, not enough information}.

On top of this learned composition we add a soft-logic prior in LOREN's style. It is extended with arc-specific operators: negation swaps supported and refuted, and a conditional antecedent that is not supported yields not enough information rather than refuted. The root's predicted distribution is regularized toward what these operators imply from the children. That regularization is what makes the node states a faithful account of the verdict.

Teacher signal. For each subtree we produce a sub-claim. This can be done by linearizing the subtree with its governing predicate, or by asking the teacher to phrase it as a standalone claim. A large instruction-tuned model grades each sub-claim against the evidence, giving a soft distribution $t_v$.

Teacher hallucination is a known risk in instruction distillation. We therefore keep a claim's node labels only when the teacher's root judgment agrees with the gold label. We also down-weight nodes whose teacher grades violate the arc operators.

The training loss is

$$\mathcal{L} = \mathrm{CE}(q_{\text{root}}, y) + \lambda \sum_{v} w_v\, D_{KL}(t_v \,\|\, q_v) + \mu\, \mathcal{L}_{\text{logic}},$$

where $y$ is the gold label, $w_v$ is the node's filter weight, and $\mathcal{L}_{\text{logic}}$ is the soft-logic consistency term.

Analysis. We will probe three things:

  • whether the student learns the flipping behavior of negation arcs;
  • how accuracy degrades as we inject parser errors;
  • whether the node states are faithful, tested by intervening on one node and checking whether the root changes as the operators predict.

Baselines

Naive reference points. The do-nothing reference is the majority class together with a claim-only classifier. The claim-only model matters because FEVER reports 50.91% accuracy when evidence is ignored. Any gain on our syntactic subset must clear this shortcut level.

Strongest existing method. The strongest competitor is LOREN, since it is the published system that already delivers faithful phrase-level verdicts with distillation on FEVER. It reports a 72.93 FEVER score on the blind test set with RoBERTa-large, over 96% agreement, and 74.3 culprit-phrase F1. We will rerun it from the released code on our splits with identical retrieved evidence. We will run it at both RoBERTa-large and at our student's encoder size, and match the hyperparameter search budget.

Beating LOREN means one of two outcomes:

  • equal overall FEVER score with a significant gain on the syntactic subset and on Symmetric FEVER; or
  • equal accuracy at a much smaller size, together with culprit F1 at least 74.3 on the same annotation protocol.

Key ablation: LOREN-dep. This baseline isolates recursion from decomposition. It takes dependency subtrees as LOREN's chunks, uses the same teacher grades, and keeps flat AND/OR aggregation. It tells us whether the gain comes from better units or from composing along arcs.

Other ablations. A no-teacher tree student, trained only on claim labels, is essentially AttentiveTreeNet transplanted to FEVER. It isolates the node-level distillation. A label-smoothing control replaces teacher grades with uniform smoothing on nodes, which addresses the label-smoothing critique of distillation.

Flat distilled student. This baseline follows Mimic and Conquer: a sequential encoder of our student's size is distilled from the same teacher's claim-level outputs, and its explanations come from post-hoc token attribution. It is the fair test of whether keeping the tree costs accuracy.

Few-shot prompting and the teacher. QA-NatVer is run as reported, using Flan-T5 at 70.3% few-shot FEVER accuracy. We also report the teacher itself prompted on whole claims. Together these bound what inference-time prompting achieves without a trained student.

Why not just…?

Why not just feed dependency-based chunks into LOREN? This is the most serious threat. It may capture most of the benefit if LOREN's failures come from bad chunk boundaries rather than flat combination. LOREN-dep is exactly this system. The prediction that separates the two is that LOREN-dep will fail specifically on negated and conditional claims, because AND/OR gates cannot express a state flip or a conditional that yields not enough information. If LOREN-dep matches the full model on that subset, we will report that recursion is unnecessary and reframe the contribution around node-level teacher distillation.

Why not just prompt a large model for chain-of-thought explanations? Its free-text reasoning is not what computes the verdict. Generated explanations also hallucinate on unverifiable claims, as e-FEVER reports for its GPT-3 summaries. The monotonicity results show large models near chance on downward inference. The prompted-teacher baseline measures this directly on our subset. The latency comparison, with DepChecker's 72 ms per sentence as the target regime, measures the deployment cost.

Why not just distill into a fast flat model and explain it with attributions? Mimic and Conquer shows such students are accurate and robust to parse noise, so this alternative could win on accuracy. However, Data and Model Distillation could only support its explanations by inspecting attention, which is weak evidence of faithfulness. Our intervention test is applied to both systems. The tree student's node edits should change the verdict as predicted. Attribution-selected token edits on the flat model should do so far less reliably. If the flat model wins on accuracy by a wide margin, we will report the size of that trade-off rather than hide it.

Proposed Experiments

Datasets and subsets. In-domain training and evaluation use FEVER, reporting label accuracy and FEVER score. We build a syntactic stress subset from the FEVER dev set by parser-based filtering for negation arcs, conditional and concessive markers, quantifiers, and modifiers far from their heads. Each flagged claim is checked by hand on a sample. Robustness to dataset artifacts is measured on Symmetric FEVER; no reported result was found on Lacuna for an exact number there.

Transfer and logic transfer. For cross-domain transfer, we train on FEVER and test on Climate-FEVER. The reference point is the zero-shot triple system of Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs at 46.71% accuracy. To check whether the learned arc operators carry over as logic, we evaluate the student's NLI head on MED against AttentiveTreeNet's 75.7% and NeuralLog's 93.4%.

Explanation quality. We measure culprit-phrase F1 on a new set of refuted claims annotated like LOREN's 100, plus a stratum drawn from the syntactic subset. We measure LOREN-style agreement. We run the node-intervention faithfulness test.

Efficiency and parser robustness. We report wall-clock latency including parsing. We plot the accuracy curve as parser arcs are randomly corrupted at increasing rates.

Decisive checks. Two results decide the proposal. First, on the syntactic subset, the full model must beat both LOREN and LOREN-dep, which isolates arc-typed recursion. Second, the no-teacher tree student must fall clearly below the distilled one, which shows node-level distillation matters. If both hold at a small fraction of RoBERTa-large's size, the claim stands. If LOREN-dep closes the gap, recursion is not the ingredient. If the no-teacher tree matches, distillation is not.