A Semantics-Aware Approach to Automated Claim Verification

Blanca Calvo Figueras, Montse Oller, Rodrigo Agerri2022 · DOI

Abstract

The influence of fake news in the perception of reality has become a mainstream topic in the last years due to the fast propagation of misleading information. In order to help in the fight against misinformation, automated solutions to fact-checking are being actively developed within the research community. In this context, the task of Automated Claim Verification is defined as assessing the truthfulness of a claim by finding evidence about its veracity. In this work we empirically demonstrate that enriching a BERT model with explicit semantic information such as Semantic Role Labelling helps to improve results in claim verification as proposed by the FEVER benchmark. Furthermore, we perform a number of explainability tests that suggest that the semantically-enriched model is better at handling complex cases, such as those including passive forms or multiple propositions.

Introduction

Automated Claim Verification is the critical task of determining the truthfulness of a statement based on available evidence. While state-of-the-art language models like BERT have significantly advanced this field, they often treat text as a series of statistical patterns rather than structured meaning. This paper addresses a specific "complexity gap" where standard models struggle with the "who did what to whom" in linguistically complex sentences, such as those using passive voice or nested clauses.

The Complexity Gap in Fact-Checking

A key observation of this research is the disparity between academic benchmarks and real-world misinformation. The authors compared the popular FEVER dataset, which consists of human-generated claims from Wikipedia, with MultiFC, a dataset of naturally occurring claims from fact-checking websites.

Comparison of claim complexity between the FEVER and MultiFC datasets.

As shown in the distribution above, FEVER claims are statistically simpler, typically containing only 1–2 verbs. In contrast, real-world "fake news" (MultiFC) exhibits a much broader distribution of complexity, often involving multiple actions and actors that can confuse standard sequence-based models.

Methodology: Semantics-Aware BERT (SemBERT)

To bridge this gap, the authors utilize the SemBERT architecture. Unlike standard BERT, which only processes raw text, SemBERT incorporates a parallel semantic stream using Semantic Role Labeling (SRL).

The SemBERT architecture, showing parallel text and semantic streams.

SRL identifies the predicate (action) and its arguments (actors, objects, locations), providing a "semantic map" that remains consistent regardless of sentence structure. For instance, in the sentence "The book was written by John," SRL correctly identifies "John" as the agent (Arg0), helping the model avoid confusion caused by the passive voice.

The authors tested several variations, finding that a simplified tag set (SemBERT_tags1) performed best by reducing data sparsity. They also introduced a novel attention mechanism to reason across multiple propositions within a single piece of evidence.

Key Results and Insights

The semantically enriched model demonstrated superior performance, particularly as claim complexity increased:

  • Overall Accuracy: The best-performing model (SemBERT_tags1) achieved a label accuracy of 75.37% on the FEVER development set, outperforming the BERT baseline (73.82%).
  • Complex Case Performance: The most striking result appeared when filtering for complex claims (those with 5 or more verbs). On these difficult cases, the semantically enriched model improved accuracy by 6.5 percentage points over BERT, compared to only a 1.5-point improvement on simple claims.
  • Robustness: Through manual adversarial testing, the authors showed that the semantically enriched model maintained its correct labels while standard BERT often flipped its prediction when sentence order was changed or voice was swapped.

Conclusion

While this work significantly improves the inference stage of claim verification, limitations remain. The current FEVER benchmark may act as a bottleneck due to its inherent simplicity. Future research will explore more robust base models like RoBERTa or XLNet and focus on generating human-readable justifications for verdicts based on these underlying semantic structures.

Summary updated

Abstract

The influence of fake news in the perception of reality has become a mainstream topic in the last years due to the fast propagation of misleading information. In order to help in the fight against misinformation, automated solutions to fact-checking are being actively developed within the research community. In this context, the task of Automated Claim Verification is defined as assessing the truthfulness of a claim by finding evidence about its veracity. In this work we empirically demonstrate that enriching a BERT model with explicit semantic information such as Semantic Role Labelling helps to improve results in claim verification as proposed by the FEVER benchmark. Furthermore, we perform a number of explainability tests that suggest that the semantically-enriched model is better at handling complex cases, such as those including passive forms or multiple propositions.
BibTeX
@misc{figueras2022semantics,
  author = {Blanca Calvo Figueras and Montse Oller and Rodrigo Agerri},
  title = {{A Semantics-Aware Approach to Automated Claim Verification}},
  year = {2022},
  doi = {10.18653/v1/2022.fever-1.5},
}
Figure