QMUL-SDS at SCIVER: Step-by-Step Binary Classification for Scientific Claim Verification

Xia Zeng, Arkaitz ZubiagaNAACL 2021

TL;DR

This paper introduces a step-by-step binary classification system for scientific claim verification, outperforming baseline methods on the SCIVER benchmark. The approach breaks down the complex task into three manageable stages: abstract retrieval, rationale selection, and label prediction, demonstrating the effectiveness of a modular, sequential approach.

What Problem Does This Paper Address?

The proliferation of online information, particularly in the scientific domain, necessitates robust automated fact-checking systems. Scientific claim verification, the process of assigning a truth value to a scientific claim, is a challenging task due to the specialized nature of scientific literature and the potential for subtle misinformation. The SCIVER shared task provides a benchmark for this problem, requiring systems to not only identify relevant scientific abstracts but also pinpoint supporting evidence within those abstracts and ultimately predict the claim's veracity. Existing approaches often struggle with the complexity of this multi-stage process, and the authors aim to develop an efficient and effective system that can handle these challenges.

Methods

The QMUL-SDS team developed a step-by-step binary classification system, breaking down the SCIVER task into three sequential modules:

  • Abstract Retrieval: Instead of traditional ranking methods, they framed this as a binary classification problem. They first used TF-IDF to retrieve the top 30 most similar abstracts and then fine-tuned a BioBERT classifier to predict whether an abstract is relevant to a given claim. This approach leverages the power of pre-trained language models for relevance assessment.
  • Rationale Selection: Similar to abstract retrieval, this stage was also treated as a binary classification task. The same BioBERT classifier, fine-tuned on abstract retrieval, was further trained to identify relevant sentences (rationales) within the retrieved abstracts, given a claim-sentence pair.
  • Label Prediction: This was implemented as a two-step binary classification process. First, a RoBERTa-large model predicts whether there is "ENOUGH_INFO" or "NOT_ENOUGH_INFO." If "ENOUGH_INFO" is predicted, a second RoBERTa-large model determines if the claim is "SUPPORT" or "CONTRADICT." This two-step approach was designed to mitigate issues with imbalanced data, particularly for the "CONTRADICT" class.

Datasets

  • SCIFEST: This dataset was used for training and evaluation throughout the paper. It consists of scientific claims paired with abstracts and rationales, requiring systems to perform abstract retrieval, rationale selection, and label prediction.

Key Insights and Ideas

  • Binary Classification for Retrieval: Framing abstract retrieval as a binary classification task, rather than a ranking problem with manual thresholds, proved more effective and made better use of available training data.
  • BioBERT for Domain Specificity: Utilizing BioBERT, a language model pre-trained on biomedical text, provided a significant advantage for scientific claim verification tasks, aligning well with the domain of the SCIVER dataset.
  • Two-Step Label Prediction: The two-step label prediction strategy (first "ENOUGH_INFO" vs. "NOT_ENOUGH_INFO," then "SUPPORT" vs. "CONTRADICT") showed improved performance, particularly in handling the challenging "CONTRADICT" class, without requiring additional data.
  • Pipeline vs. End-to-End: The paper highlights that while modular, step-by-step approaches can be effective, errors can accumulate. This suggests that future work might benefit from more integrated or multi-task learning approaches.

Key Results

  • The QMUL-SDS system achieved the 6th position overall and 4th among distinct teams on the SCIVER leaderboard.
  • On the dev set, their system achieved substantial improvements over the baseline, with precision scores 15.75%-23.37% higher, recall scores 3.82%-14.21% higher, and F1 scores 10.11%-16.08% higher.
  • The two-step label prediction method achieved an accuracy of 85.98% and a weighted-F1 of 85.84% on the dev set with oracle abstracts and rationales, outperforming the baseline's 81.93% accuracy and 81.85% weighted-F1.
  • The abstract retrieval module, using BioBERT-large, achieved a recall of 74.16%, a significant improvement over the baseline's 69.86%.

Limitations and Future Work

  • Error Accumulation: The authors acknowledge that errors in earlier modules of the pipeline can propagate and affect downstream performance.
  • Neutral Detector Robustness: The "neutral detector" in the two-step label prediction module was found to be less robust when operating within the full pipeline compared to using oracle evidence. Future work could involve training this detector on evidence retrieved by previous modules to improve its pipeline performance.
  • Systematic Design: The paper suggests that a more systematic system design, potentially involving multitask learning to optimize all subtasks simultaneously, could lead to further improvements.
  • Model Size vs. Performance: While larger models like BioBERT-large showed improvements in recall for abstract retrieval, they sometimes suffered from precision drops, indicating a need for careful model selection and tuning.

Summary updated

BibTeX
@inproceedings{zeng2021qmul,
  author = {Xia Zeng and Arkaitz Zubiaga},
  title = {{QMUL-SDS at SCIVER: Step-by-Step Binary Classification for Scientific Claim Verification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure