Evidence-Aware Hierarchical Interactive Attention Networks for Explainable Claim Verification

Lianwei Wu, Yuan Rao, Xiong Yang, Wanzhen Wang, Ambreen NazirIJCAI 2020 · OpenReview

TL;DR

This paper introduces EHIAN, a novel neural network architecture for explainable claim verification that goes beyond simply identifying relevant articles. EHIAN focuses on pinpointing specific evidence fragments within articles that directly address the questionable aspects of a claim, while also incorporating source credibility to filter out unreliable information, leading to more accurate and interpretable verification results.

What Problem Does This Paper Address?

The proliferation of unverified claims, hoaxes, and fake news on social media poses a significant challenge. While automatic claim verification methods exist, many struggle to provide clear explanations for their decisions. Existing evidence-based approaches often capture broad evidence (e.g., top articles or salient words) which may not directly address the core of a claim's veracity. Furthermore, these methods often neglect the crucial aspect of source credibility, leading to potentially unreliable evidence interfering with the verification process. This paper aims to bridge this gap by developing a system that not only verifies claims but also explains why a claim is deemed true or false by focusing on specific, credible evidence fragments.

Methods

The proposed Evidence-aware Hierarchical Interactive Attention Networks (EHIAN) is a four-level architecture:

  1. Input Embedding Layer: Utilizes pre-trained BERT models to generate token embeddings for claims and their associated relevant articles. It also computes an average embedding of all relevant articles (X_avg) to serve as a global semantic benchmark.
  2. Internal Interaction Layer (IIL): This layer is designed to capture key evidence fragments. It employs a combination of multi-head self-attention and symmetrical attention mechanisms to facilitate deep interaction and matching between the claim and each individual relevant article. This helps in identifying specific parts of articles that are most relevant to the claim.
  3. Global Inference Layer (GIL): This is the core of EHIAN's explainability and credibility-aware approach. It has two main components:
  4. Gated Affine Transformation: This novel technique fuses source credibility features (derived from metadata like domain suffix, backlinks, traffic, etc.) with the article's semantics. Instead of simple concatenation, it uses a gating mechanism to selectively incorporate credibility information, ensuring that more trustworthy sources have a greater influence.
  5. Global Interaction Inference: This component mitigates the impact of extreme or outlier semantics from individual articles. It achieves this by integrating the average semantics of all relevant articles, effectively balancing the overall evidence and preventing a single unreliable article from skewing the results.
  6. Task Learning Layer: A standard softmax layer predicts the claim's veracity, and a cross-entropy loss function is used for training.

Datasets

The paper evaluates EHIAN on two publicly available fact-checking datasets:

  • Snopes: Contains claims labeled as "true" or "false."
  • PolitiFact: Originally has six credibility labels, which are merged into "true," "false," and "mixed" for this study.

Both datasets include news claims, their credibility labels, sets of relevant articles, and source website domains. The paper notes that source credibility features were derived from Alexa data due to the original datasets not containing this information.

Key Insights and Ideas

  • Focusing on Questionable Parts: The most significant insight is the shift from capturing general evidence to identifying specific "evidence fragments" that directly address the questionable aspects of a claim. This is crucial for providing meaningful explanations.
  • Source Credibility as a Filter: Explicitly incorporating source credibility is vital. By using metadata to assess the trustworthiness of article sources, EHIAN can effectively down-weight or ignore information from unreliable outlets, preventing them from polluting the verification signal.
  • Gated Affine Transformation for Fusion: The novel gated affine transformation provides a more sophisticated way to fuse source credibility with article semantics than simple concatenation or addition. This allows the model to dynamically decide how much influence credibility information should have.
  • Global Interaction for Robustness: Mitigating the impact of extreme semantics by considering the average semantics of all articles is key to building a robust system that isn't easily swayed by outlier information.

Key Results

EHIAN achieves state-of-the-art performance on both the Snopes and PolitiFact datasets.

  • Performance Boost: EHIAN outperforms several strong baselines, including DeClarE, HAN, and HAN-ba, by a notable margin. For instance, on the Snopes dataset, EHIAN achieves a micF1 score of 0.831 for "True" claims and 0.893 for "False" claims, surpassing previous methods.
  • Ablation Studies: The ablation studies confirm the effectiveness of each component:
  • The Internal Interaction Layer (IIL) provides a significant boost (around 5-7% micF1) over models without it.
  • The Global Inference Layer (GIL) also contributes substantially (around 6-9% micF1).
  • The Gated Affine Transformation shows a 2-3% improvement in macF1.
  • Global Interaction Inference provides a smaller but consistent boost (1-2% macF1).
  • Explainability: Visualizations demonstrate that EHIAN successfully captures specific keywords and phrases from credible sources that directly relate to the claim's veracity, providing clear evidence for its predictions.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that deriving source credibility features relied on external data (Alexa), which might not always be available or perfectly representative.
  • The model's performance is dependent on the quality and relevance of the provided articles.
  • Future Work: The authors plan to enhance inference to discover conflicts between claims and relevant articles, suggesting a direction towards more nuanced understanding of evidence. They also aim to expand the work, implying potential for broader applications or more complex claim verification scenarios.

Summary updated

Abstract

Exploring evidence from relevant articles to confirm the veracity of claims is a trend towards explainable claim verification. However, most strategies capture the top-k check-worthy articles or salient words as evidence, but this evidence is difficult to focus on the questionable parts of unverified claims. Besides, they utilize relevant articles indiscriminately, ignoring the source credibility of these articles, which may cause quiet a few unreliable articles to interfere with the assessment results. In this paper, we propose Evidence-aware Hierarchical Interactive Attention Networks (EHIAN) by considering the capture of evidence fragments and the fusion of source credibility to explore more credible evidence semantics discussing the questionable parts of claims for explainable claim verification. EHIAN first designs internal interaction layer (IIL) to strengthen deep interaction and matching between claims and relevant articles for obtaining key evidence fragments, and then proposes global inference layer (GIL) that fuses source features of articles and interacts globally with the average semantics of all articles and finally earns the more credible evidence semantics discussing the questionable parts of claims. Experiments on two datasets demonstrate that EHIAN not only achieves the state-of-the-art performance but also secures effective evidence to explain the results.
BibTeX
@inproceedings{wu2020evidence,
  author = {Lianwei Wu and Yuan Rao and Xiong Yang and Wanzhen Wang and Ambreen Nazir},
  title = {{Evidence-Aware Hierarchical Interactive Attention Networks for Explainable Claim Verification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=shTqWL74gYW},
}
Figure