Multiple Evidence Combination for Fact-Checking of Health-Related Information

Pritam Deka, Anna Jurek-Loughrey, Deepak PACL 2023

Sure, here's a blog-style summary of the paper "Multiple Evidence Combination for Fact-Checking of Health-Related Information":

TL;DR

This paper tackles the challenge of automatically verifying health-related claims by proposing a domain-specific BERT model trained on natural language inference (NLI) data. It also introduces a novel method for combining multiple, potentially conflicting, pieces of evidence to arrive at a final veracity judgment for a claim.

What Problem Does This Paper Address?

In the digital age, misinformation, especially concerning health, spreads rapidly. Manually fact-checking these claims is time-consuming and requires domain expertise. While automated fact-checking methods exist, they often struggle with situations where multiple pieces of evidence for a single claim might contradict each other. This paper aims to address this by developing a robust system that can not only classify individual claim-evidence pairs but also synthesize information from multiple sources, even when they present conflicting views, to provide a final verdict on the claim's truthfulness.

Methods

The core of the proposed solution involves two main components:

  • Claim-Evidence Pair Classification: The authors frame this as a Natural Language Inference (NLI) task. They leverage a domain-specific BERT model, PubMedBERT, and pre-train it on the Multi-Genre Natural Language Inference (MultiNLI) dataset. This pre-trained model is then further fine-tuned on MedNLI, a biomedical NLI dataset, to enhance its performance on health-related claims. This approach allows the model to learn to classify a claim-evidence pair as "support," "refute," or "neutral."
  • Multiple Evidence Combination: To handle conflicting evidence, the paper employs an improvised Dempster-Shafer (D-S) theory of evidence combination. This theory is well-suited for situations where evidence might be uncertain or contradictory. The authors adapt the D-S theory to mitigate issues like zero denominators that can arise with conflicting evidence, ensuring a more stable combination process.

Additionally, the paper explores data augmentation techniques, specifically back-translation using Google Translate and transformer-based NMT models, to improve model performance, especially in low-data scenarios.

Datasets

The primary dataset used for evaluation is SCIFACT. The authors also utilize:

  • MultiNLI (Multi-Genre Natural Language Inference): Used for initial pre-training of the PubMedBERT model.
  • MedNLI (Biomedical NLI): Used for further fine-tuning the model on domain-specific NLI data.
  • HEALTHVER and COVIDFACT: Used for evaluating the model's generalization capabilities on other biomedical fact-checking datasets.

Key Insights and Ideas

  • NLI as a Foundation for Fact-Checking: Framing claim-evidence pair classification as an NLI task proves effective, especially when using domain-specific pre-trained models like PubMedBERT. This suggests that general language understanding capabilities can be effectively transferred to specialized fact-checking domains.
  • Dempster-Shafer Theory for Conflicting Evidence: The Dempster-Shafer theory offers a principled way to combine evidence from multiple sources, even when they disagree. The authors' improvisation on this theory helps overcome practical limitations and provides a more robust solution for synthesizing information.
  • Data Augmentation is Crucial: Back-translation, particularly using transformer-based NMT models, significantly improves performance, especially when labeled data is scarce. This highlights the importance of data augmentation for building effective fact-checking systems.
  • Domain Specificity Matters for Generalization: Models trained on specific biomedical datasets (like SCIFACT) tend to generalize better to other similar biomedical datasets (like HEALTHVER) than to datasets containing a mix of scientific and non-scientific claims (like COVIDFACT). This underscores the need for diverse training data or domain-adaptive techniques for broader applicability.

Key Results

  • The proposed PubMedBERT model, fine-tuned on MedNLI, achieved strong performance on the SCIFACT dataset, outperforming other models in zero-shot settings.
  • The D-S theory-based combination method, when applied to the SCIFACT dev set, yielded promising results with precision, recall, and F1 scores around 0.89.
  • Data augmentation via back-translation led to significant improvements in performance, with transformer-based NMT models showing better results than Google Translate.
  • The model demonstrated good generalization between SCIFACT and HEALTHVER, but less so to COVIDFACT, indicating the impact of data domain on transfer learning.

Limitations and Future Work

  • Data Scarcity: The paper acknowledges that datasets like SCIFACT have limited labeled data, which can hinder model performance. Data augmentation is presented as a solution, but further research into more effective augmentation techniques is suggested.
  • Generalization to Non-Scientific Claims: The model's performance degrades when tested on datasets containing non-scientific claims, suggesting a need for models that can handle both scientific and non-scientific misinformation.
  • Future Work: The authors are interested in exploring prompt-based methods and incorporating multimodal data (e.g., images, videos) into their fact-checking approaches. They also aim to investigate how to build models that can generalize better across diverse types of misinformation.

Summary updated

BibTeX
@inproceedings{deka2023multiple,
  author = {Pritam Deka and Anna Jurek-Loughrey and Deepak P},
  title = {{Multiple Evidence Combination for Fact-Checking of Health-Related Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure