Automatic Claim Review for Climate Science via Explanation Generation

Shraey Bhatia, Jey Han Lau, Timothy BaldwinarXiv 2021 · arXiv

Introduction

The digital landscape is increasingly saturated with climate change skepticism and misinformation, posing a significant hurdle to public understanding and effective political action. While specialized platforms like Climate Feedback and Skeptical Science offer rigorous, expert-led rebuttals, the sheer volume of online claims creates a bottleneck for manual review. Traditional automated fact-checking systems often fall short by providing only a binary "veracity label" (e.g., Supported vs. Refuted) without the context necessary to persuade or inform. This research introduces a system designed for Automatic Claim Review, which not only predicts the truthfulness of a claim but also generates a scientifically grounded, evidence-based explanation.

Methodology: Fusion-in-Decoder

The researchers framed fact-checking as an open-domain question-answering task, utilizing a Fusion-in-Decoder (FiD) architecture. This approach moves beyond simple classification by synthesizing information from multiple external sources to justify a verdict.

The Fusion-in-Decoder (FiD) architecture for automatic claim review, combining retrieval and generative components.

The system operates in two distinct phases:

  1. Retrieval: For any given claim, the system queries an external knowledge base. The study compared a traditional keyword-based algorithm (BM25) with a modern neural approach called Binary Passage Retrieval (BPR).
  2. Generation: The claim and the top retrieved passages are processed by a T5 (Text-to-Text Transfer Transformer) model. The "Fusion" mechanism allows the decoder to attend to all retrieved passages simultaneously, enabling it to generate a single, cohesive explanation followed by a veracity label.

Data and Knowledge Sources

The model was trained and evaluated using two primary datasets:

  • CLIMATE-FEVER: A large-scale dataset containing 1,535 claims with evidence sourced from Wikipedia.
  • FEEDBACK: A smaller but high-quality dataset of 130 instances crawled from climatefeedback.org. This dataset is particularly valuable as it contains "Key Takeaways" written by scientists, serving as the "gold standard" for professional scientific communication.
An example of the expert-written reviews from the FEEDBACK dataset, showing the claim, verdict, and detailed scientific explanation.

The researchers also compared the effectiveness of using general knowledge from Wikipedia versus domain-specific data from PUBS (a collection of PubMed abstracts and IPCC reports).

Key Insights and Results

The study yielded several critical insights into the automation of scientific fact-checking:

  • Knowledge Integration: Incorporating external evidence significantly improved veracity prediction accuracy. The best-performing model (using Wikipedia and BPR) achieved 80% accuracy on the FEV2 dataset, compared to 75% for a "closed-book" T5 model.
  • Retrieval Performance: In the specialized context of climate science, the neural BPR retriever outperformed BM25 when retrieving a limited number of documents, highlighting the importance of semantic context over simple keyword matching.
  • Explanation Quality: Using BERTScore to measure semantic similarity, the FiD approach consistently produced higher-quality justifications than baseline models. When trained on the expert-led FEEDBACK dataset, the model successfully adopted scientific terminology (e.g., "pH levels," "species impact"), moving away from simple verbatim copying toward more sophisticated synthesis.

Limitations and Future Directions

Despite its promise, the system faces challenges. The FEEDBACK dataset is heavily biased toward "Incorrect" claims, making it difficult to train the model on explaining why a claim is correct. Furthermore, like all generative transformers, there is a persistent risk of "hallucination"—generating text that sounds authoritative but is scientifically inaccurate.

While the system is not yet a replacement for human experts, it demonstrates significant potential as a "drafting assistant," helping scientists and fact-checkers generate initial rebuttals more efficiently. Future work will likely focus on bridging the stylistic gap between general-purpose knowledge bases and specialized scientific discourse.

Summary updated

BibTeX
@misc{bhatia2021automatic,
  author = {Shraey Bhatia and Jey Han Lau and Timothy Baldwin},
  title = {{Automatic Claim Review for Climate Science via Explanation Generation}},
  year = {2021},
  eprint = {2107.14740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.14740},
}
Figure