Multi-Evidence Based Fact Verification via A Confidential Graph Neural Network

Yuqing Lan, Zhenghao Liu, Yu Gu, Xiaoyuan Yi, Xiaohua Li, Liner Yang, Ge Yu

In an era of rampant misinformation, automated fact-checking is no longer a luxury—it’s a necessity. However, teaching AI to verify a claim isn't just about finding the right information; it’s about ignoring the wrong information. A new paper by Yuqing Lan and colleagues introduces CO-GAT, a "Confidential" Graph Attention Network designed to prevent irrelevant evidence from "polluting" the reasoning process.

TL;DR

CO-GAT introduces a node-masking mechanism that calculates a confidence score for each piece of evidence. By effectively "muting" irrelevant or noisy evidence before it can influence other nodes in a reasoning graph, the model achieves a 73.59% FEVER score and demonstrates superior performance in both general and scientific domains.

What Problem Does This Paper Address?

Most modern fact-verification systems follow a three-step pipeline: retrieve relevant documents, select specific evidence sentences, and then perform reasoning to reach a verdict (Support, Refute, or Not Enough Info).

The "reasoning" step often uses Graph Attention Networks (GATs). In these models, each claim-evidence pair is a node, and every node is connected to every other node. While this allows for deep interaction between different pieces of evidence, it creates a "noise propagation" problem. If the retrieval step pulls in three relevant sentences and two irrelevant ones, the irrelevant "noisy" nodes can bleed their semantics into the "golden" nodes during the graph's message-passing phase. This leads to diluted representations and incorrect factual verdicts.

Methods

The authors propose the Confidential Graph Attention Network (CO-GAT) to act as a semantic filter. The architecture consists of three main stages:

  1. Node Encoding: The model uses Pre-trained Language Models (PLMs) like ELECTRA or RoBERTa to encode claim-evidence pairs into initial vector representations.
  2. Node Masking (The Core Innovation): Before the graph reasoning begins, the model calculates a Confidence Score (CO-SCO) for each node. This score estimates the relevance of the evidence to the claim.
  3. If the confidence is high, the node retains its original information.
  4. If the confidence is low, the node is masked by blending it with a "blank node" (a representation of the claim without any evidence).
  5. This "denoising" ensures that only trustworthy information flows through the graph.
  6. Graph Reasoning: The denoised representations are fed into a standard GAT. Because the noisy nodes have been "muted," the attention mechanism can focus more sharply on the "golden" evidence.

The model is trained using a multi-task learning strategy, optimizing for both the final claim verdict and the intermediate node confidence scores simultaneously.

Datasets

The researchers evaluated CO-GAT on two major benchmarks:

  • FEVER: A large-scale dataset based on Wikipedia, containing over 185,000 claims. It is the standard benchmark for general-domain fact verification.
  • SCIFACT: A specialized dataset containing 1,409 claims sourced from scientific abstracts. This tests the model’s ability to generalize to complex, technical language.

Key Insights and Ideas

  • The "Blank Node" Strategy: Instead of simply deleting a node (which might break the graph structure), the authors replace noisy evidence with a "blank node" containing only the claim. This effectively tells the model: "There is no useful evidence here; don't let this node influence your neighbors."
  • Better Calibration with NEI: One of the most interesting findings is that CO-GAT is much better at predicting "NOT ENOUGH INFO" (NEI). Standard models often try to force a "Support" or "Refute" verdict even when the evidence is garbage. CO-GAT’s masking mechanism allows it to recognize when the available evidence is insufficient.
  • Concentrated Attention: By visualizing the attention weights, the authors showed that CO-GAT produces a more "concentrated" edge attention distribution. In other words, the model becomes much more decisive about which evidence pieces actually matter.

Key Results

  • FEVER Performance: CO-GAT achieved a 73.59% FEVER score on the blind test set, outperforming several graph-based baselines like GEAR and KGAT.
  • Ablation Success: Removing the node masking mechanism (reverting to a standard MHA-GAT) resulted in a significant performance drop, proving that the "confidential" filtering is the primary driver of the model's success.
  • Cross-Domain Generalization: The model showed strong performance on SCIFACT, suggesting that the logic of filtering noise is universal and not just overfitted to Wikipedia-style text.

Limitations and Future Work

The authors acknowledge a potential "hard-label" risk: if the confidence scorer makes a mistake and masks a piece of "golden" evidence, that information is lost forever to the reasoning engine. To mitigate this, they use a "soft score" (a probability between 0 and 1) rather than a binary mask, but the risk of erasing critical information remains.

Future work could involve more sophisticated ways of generating the "blank node" or integrating external knowledge graphs to help the confidence scorer make better initial judgments about evidence relevance.

Summary updated

Abstract

Fact verification tasks aim to identify the integrity of textual contents according to the truthful corpus. Existing fact verification models usually build a fully connected reasoning graph, which regards claim-evidence pairs as nodes and connects them with edges. They employ the graph to propagate the semantics of the nodes. Nevertheless, the noisy nodes usually propagate their semantics via the edges of the reasoning graph, which misleads the semantic representations of other nodes and amplifies the noise signals. To mitigate the propagation of noisy semantic information, we introduce a Confidential Graph Attention Network (CO-GAT), which proposes a node masking mechanism for modeling the nodes. Specifically, CO-GAT calculates the node confidence score by estimating the relevance between the claim and evidence pieces. Then, the node masking mechanism uses the node confidence scores to control the noise information flow from the vanilla node to the other graph nodes. CO-GAT achieves a 73.59% FEVER score on the FEVER dataset and shows the generalization ability by broadening the effectiveness to the science-specific domain.
BibTeX
@article{lan2025multi,
  author = {Yuqing Lan and Zhenghao Liu and Yu Gu and Xiaoyuan Yi and Xiaohua Li and Liner Yang and Ge Yu},
  title = {{Multi-Evidence Based Fact Verification via A Confidential Graph Neural Network}},
  journal = {IEEE Trans. Big Data},
  year = {2025},
  doi = {10.1109/tbdata.2024.3403382},
}
Figure