Systems that check claims against documents work in two steps. They find evidence passages, then decide whether those passages support or refute the claim. Both steps break down when the claim contains a negation such as "X never held title Y" or "the film by A that did not win award Z." A useful observation is that the evidence needed for a negated claim is the same evidence needed for its positive twin. Whether or not A won Z, you need the passage about A and Z. So the evidence-finding step should ignore polarity entirely, and only the final verdict should depend on it. We propose to build verifiers on that rule. A decomposer rewrites each claim into positive atomic facts, each tagged with a polarity bit and its place in the claim structure. A retriever and verifier are trained together, and they only ever see positive facts. Negation is applied symbolically after the evidence for each fact has been judged and before the facts are combined. Because negation tokens never reach the neural parts, the "pink elephant" failure cannot occur.
The second half of the proposal concerns how to tell whether any of this matters, given that public fact-checking benchmarks are both memorized and full of shortcuts. We build test sets from FEVER, HoVer and Wikidata. They contain negated twins, whose labels follow by rule from an original claim, and fresh compositions whose strings appear nowhere on the web. A model that memorized a claim, or learned that "not" predicts REFUTES, scores well on the original and poorly on its twin. The gap between the two scores is the measurement.
Core Claim
Our hypothesis is that evidence retrieval should be polarity-invariant and verdicts polarity-equivariant. Systems that enforce this through the architecture will outperform both single joint models and agentic LLM pipelines on claims with negation scoped to one step of a multi-step chain. They will also lose almost nothing on positive claims. Let $c$ be a claim and $\neg_i c$ the same claim with the atomic fact at position $i$ negated. Let $E(\cdot)$ be the retrieved evidence set and $v(\cdot)$ the verdict. Then the property we want is
$$ E(\neg_i c) = E(c), \qquad v(\neg_i c) = \mathrm{Compose}\big(v(a_1),\dots,\neg v(a_i),\dots,v(a_n)\big), $$
where $a_1,\dots,a_n$ are the positive atomic facts of $c$ and Compose is the conjunction or chain structure the decomposer recovers. We also claim that much of the reported accuracy on negated claims is not real compositional verification. The paired twin test will expose this, separately from any new method.
Why This Combination, and What Is Already Done
Several existing results motivate the design. Logical Consistency is Vital: Neural-Symbolic Information Retrieval for Negative-Constraint Queries (NS-IR) shows that dense retrievers return documents containing the very concept a query excludes, and repairs this with a reranker that uses logic formulas. On its NegConstraint benchmark it raises mean average precision from 36.3 (BGE) to 53.3. How Language Models Process Negation finds that middle layers of Llama-3.1-8B correctly build the opposite concept, but late-layer heads that follow word co-occurrence override it. Removing those heads lifts accuracy on negated prompts from 50.5% to 67.8%. Counterfactual Debiasing for Fact Verification (CLEVER) shows that FEVER-trained verifiers learn "negation word implies REFUTES" from the claim alone. Taken together, negation inside the neural pathway is a liability at both the evidence step and the verdict step.
One paper supplies the main design principle. LLM-MatLogic: Executable Exchange Contracts for Knowledge-Graph Query Answering with Scoped Negation proves that on knowledge graphs, negation must be scheduled in a fixed order: produce evidence, then negate, then compose. It also shows that the language model translating the query into constraints is the bottleneck, since accuracy jumps when the true constraints are supplied. That work lives entirely in closed-world graphs and does not train its translator jointly with anything. We carry its scheduling rule to unstructured text, where the "executor" is a learned retriever and verifier, not a matrix engine.
Two lines of work supply components that we reuse:
- Joint retrieval-verification. Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction treats evidence relevance as a hidden variable inside one DeBERTaV3 model, reaching a FEVER score of 76.5. Its authors concede it lags iterative systems on multi-hop HoVer. Refining Attention for Explainable and Noise-Robust Fact-Checking with Transformers (ATTUN) co-trains evidence-usefulness labels with the verdict. Neither looks at negated claims.
- Multi-agent decomposition. Fact in Fragments: Deconstructing Complex Claims via LLM-based Atomic Fact Extraction and Verification (AFEV) decomposes claims into atomic facts step by step and reaches 78.87% label accuracy on HoVer. Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checking (WKGFC) reaches 72.8% on HoVer with a graph-plus-web agent. Both pass negated atoms straight to an LLM verifier, and neither trains its retriever and verifier together.
What we take from these is a decomposer that tracks polarity, plus a Claim-Dissector-style joint model trained only on positive facts. What we do differently from all of them is make the polarity bit explicit and route it around the neural components.
DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks constrains the evaluation side. It generates fresh reasoning problems from graphs, and on these, models that look strong on static benchmarks collapse. It works only for symbolic tasks, and the file names extending this approach to evidence-grounded verification as unsolved. Our twin construction is a narrow, rule-labeled version of that idea for fact checking.
Research Program
Benchmark construction. The test sets come from three sources, and each has different label guarantees:
- FEVER twins. We take FEVER claims and negate the main predicate. SUPPORTED becomes REFUTED and the reverse; NOT ENOUGH INFO stays the same. The gold evidence stays unchanged, which also tests the invariance claim directly.
- HoVer twins. HoVer's Not-Supported label mixes "contradicted" with "insufficient," so we negate only Supported claims. The negated hop then makes the twin refuted by the same evidence chain.
- Wikidata compositions. We sample multi-hop paths and insert negation at a chosen hop. Labels come from executing the query in the produce-negate-compose order. The claim is written out in text, and the matching Wikipedia passages serve as evidence.
The Wikidata set needs care because a knowledge graph is not a complete record. "No award-received edge" is weak evidence that the award was never received. We therefore negate only attributes that are single-valued or drawn from complete lists. We spot-check a sample of a few hundred claims by hand and report the label error rate.
Every negated twin has two controls:
- a meaning-preserving paraphrase of the original, which separates sensitivity to rewording from sensitivity to negation;
- a "decoy" variant that adds a negation word outside the scope that matters, so the label is unchanged. This catches the CLEVER-style shortcut.
System. There are three components:
- Decomposer. An instruction-tuned open model outputs positive atomic facts, polarity bits, and the composition structure. It is trained on synthetic negated or positive pairs whose decomposition is known by construction.
- Retriever. A dense bi-encoder.
- Verifier. A Claim-Dissector-style joint reranker and verdict model over positive atoms. The retriever is trained to match the verifier's hidden per-passage relevance through a divergence loss, so verdict quality directly shapes what gets retrieved.
Polarity is applied after each atom's verdict and before composition. Because negated atoms are never seen in training, the retriever and verifier are trained on ordinary FEVER and HoVer data plus the decomposed atoms.
Analysis. For each system we measure two kinds of failure:
- Retrieval drift: how much the retrieved evidence for a twin differs from the evidence for its original, measured as overlap between the two top-$k$ sets.
- Verdict errors. These are split into three types. A polarity error is a correct original verdict with a twin verdict that was not flipped. A scope error is negating the wrong hop. A decoy error is flipping when the label should not change.
Baselines
Naive references. There are two:
- A plain retrieve-then-read pipeline: BGE retrieval feeding a DeBERTaV3 verifier fine-tuned on FEVER and HoVer, with the claim text passed through unchanged.
- A claim-only verifier, which sees no evidence and estimates how far shortcuts alone go.
The original FEVER paper reports that its pipeline reached 31.87% with evidence and 50.91% when ignoring evidence. That gap is a reminder of how much claim-only signal exists. On twins with balanced negation we expect the claim-only model to fall to chance; if it does not, our twin generator itself leaks the label.
Strongest LLM-agent competitor. This is AFEV (Fact in Fragments), which reports 78.87% label accuracy on HoVer. That is the highest HoVer figure in the file; WKGFC reports 72.8%. AFEV is the strongest competitor because it already decomposes claims into atoms. The only difference from our system is that negated atoms stay in the text it hands to the LLM verifier, and its retriever and verifier are trained separately.
We re-run AFEV with the same backbone LLM size as our decomposer, the same Wikipedia corpus and the same top-$k$. Prompts are tuned on the same development split with an equal budget of prompt iterations. Beating it means two things together: higher accuracy on scoped-negation twins with non-overlapping bootstrap confidence intervals, and staying within one point of AFEV's accuracy on original HoVer claims under our reproduction. We compare to our reproduction, not to the 78.87 figure, because the corpus and backbone may differ.
Strongest joint model. This is Claim-Dissector, which reports a FEVER score of 76.5. We train it on the same data as our verifier. We also add CLEVER's subtraction of the claim-only logits on top of it, since that is the established fix for negation shortcuts. CLEVER reports gains of 17.55 points on FEVER-Symmetric over BERT-base.
Strongest retrieval fix. Here the BGE retriever is replaced with an NS-IR-style logic-aware reranker. NS-IR reports MAP 53.3 versus 36.3 for BGE on NegConstraint. We measure this end to end, not just on retrieval.
Ablations isolating the key ingredient. Four variants of our system:
- Negation left in the atom text (factoring removed, all else equal).
- Factoring kept, but the retriever and verifier trained separately.
- A whole-claim flip: strip any negation, verify the positive claim, invert the label. This variant applies polarity at the claim level, not per hop.
- A joint model with no decomposition, trained on negation-augmented data with a consistency loss that enforces the invariance and equivariance property directly.
Why not just…?
Why not just strip the negation and flip the label? This is the most serious threat. On FEVER twins it should work almost perfectly, and we expect it to match our system there. It breaks only when negation is scoped to one hop, because then the verdict is not the inverse of the positive claim's verdict. On Wikidata compositions with negation at an inner hop, the whole-claim flip should make systematic scope errors. Suppose it does not, because real claims rarely have inner-hop negation. Then the honest conclusion is that a two-line heuristic suffices for current data, and our contribution reduces to the measurement study. We will report the share of naturally occurring negations in AVeriTeC and LIAR-New that fall at inner hops, so readers can judge how much the scoped case matters.
Why not just train a single joint model on negation-augmented data? Augmentation plus a consistency loss is the cheaper route to the same property, and it avoids the cost of a decomposer. The evidence against it is indirect. The negation-mechanism study shows that late-layer co-occurrence heads override correct middle-layer computation, which suggests fine-tuning may suppress the shortcut on the training distribution without removing it. The augmentation ablation settles this. If it matches the factored system on the fresh Wikidata compositions, not just on FEVER twins, we drop the architectural claim and recommend augmentation.
Why not just prompt a frontier model with step-by-step reasoning? We include a large model with chain-of-thought prompting as an additional reference. The file's evidence suggests structure still matters. DyVal reports that GPT-4 drops 13.6% on reachability when the description order is reversed, and that 33.8% of its failures are verdicts with no supporting reasoning. A large model's knowledge of the original FEVER claims is also exactly what the twin gap is designed to detect. We therefore report its accuracy on the original-minus-twin pairs, not on originals alone.
Why not just put a logic-aware reranker in front of a standard verifier? NS-IR fixes the retrieval step for exclusion queries. But for verification, the passage it would push down, the one mentioning the excluded concept, is often the decisive evidence. "A did not win Z" is refuted precisely by the passage saying A won Z. Our retrieval-drift measurement tests whether NS-IR-style reranking discards gold evidence on twins.
Proposed Experiments and Decisive Checks
Main comparison. All systems are trained on the same FEVER and HoVer training splits. They are evaluated on original claims, paraphrases, negated twins, decoys and Wikidata compositions. We report:
- label accuracy and macro F1;
- pair consistency: the share of original/twin pairs where both verdicts are correct;
- evidence recall at $k$ against gold evidence;
- the retrieval-drift and error-type breakdowns described above.
First decisive check: invariance. If our system's retrieved evidence for twins is nearly identical to that for the originals while every baseline shows substantial drift, the core claim holds at the retrieval step. If baselines also show little drift, retrieval is not where negation hurts, and the joint-training component should be reconsidered.
Second decisive check: the contamination gap. We compare the drop from originals to twins on FEVER and HoVer with the accuracy on fresh Wikidata compositions of matched depth. A large twin drop for LLM-based systems, together with low accuracy on fresh compositions, would indicate that reported negated-claim accuracy rests on memorization or shortcuts. A small drop would mean contamination is not driving these numbers, which is also worth publishing.
Third decisive check: the decomposer as bottleneck. We repeat the main comparison with gold decompositions supplied, echoing MatLogic's oracle-constraint experiment. If gold decompositions close most of the remaining gap, the next investment belongs in the decomposer, not in further joint training.
References
- Logical Consistency is Vital: Neural-Symbolic Information Retrieval for Negative-Constraint Queries
- How Language Models Process Negation
- Counterfactual Debiasing for Fact Verification
- LLM-MatLogic: Executable Exchange Contracts for Knowledge-Graph Query Answering with Scoped Negation
- Claim-Dissector: An Interpretable Fact-Checking System with Joint Re-ranking and Veracity Prediction
- Refining Attention for Explainable and Noise-Robust Fact-Checking with Transformers
- Fact in fragments: Deconstructing complex claims via LLM-based atomic fact extraction and verification
- Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checking
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks