Overview and Insights from the SciVer Shared Task on Scientific Claim Verification

David Wadden, Kyle Lo

Abstract

We present an overview of the SciVer shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021. In this shared task, systems were provided a scientific claim and a corpus of research abstracts, and asked to identify which articles SUPPORT or REFUTE the claim as well as provide evidentiary sentences justifying those labels. 11 teams made a total of 14 submissions to the shared task leaderboard, leading to an improvement of more than +23 F1 on the primary task evaluation metric. In addition to surveying the participating systems, we provide several insights into modeling approaches to support continued progress and future research on the important and challenging task of scientific claim verification.

Introduction

The exponential growth of scientific literature makes manual verification of claims nearly impossible, creating a pressing need for automated fact-checking systems. Scientific claim verification is particularly challenging due to the complexity of entailment in technical writing—often requiring reasoning over numerical data and causal relationships—and the scarcity of expert-annotated training data.

The SciVer shared task, held at NAACL 2021, aimed to address these challenges. Participants were tasked with building systems that, given a scientific claim, could retrieve relevant abstracts from a corpus, identify specific evidentiary sentences (rationales), and predict whether the evidence supports or refutes the claim. This paper summarizes the methodologies, results, and key insights derived from the 14 submissions made by 11 participating teams.

System Architectures and Participation

Most successful systems adopted a modular three-stage pipeline consisting of abstract retrieval, evidence selection, and label prediction. Innovation primarily occurred through the integration of large-scale pre-trained language models and sophisticated re-ranking strategies.

Table 1: Participating systems, team affiliations, and associated research papers.

Key architectural trends included:

  • Neural Re-ranking: Moving beyond traditional BM25 or TF-IDF retrieval by using models like T5 or BERT to re-rank initial search results.
  • Context-Aware Selection: Encoding entire abstracts to provide context for sentence-level evidence selection, rather than evaluating sentences in isolation.
  • Transfer Learning: Utilizing open-domain fact-checking datasets like FEVER to initialize models before fine-tuning on domain-specific data.

Experimental Setup

The primary benchmark for the task was SCIFACT, which includes 1,409 expert-curated claims and 5,183 biomedical abstracts. Participants also leveraged external datasets:

  • FEVER: Used for large-scale pre-training.
  • CORD-19: Utilized for domain-specific language model adaptation in the context of COVID-19 research.

Results

The shared task resulted in substantial improvements over existing baselines. The VerT5erini system achieved the top sentence-level F1 score of 63.40, representing a +23.87 F1 improvement over the VeriSci baseline. At the abstract level, the ParagraphJoint system reached an F1 of 67.16.

Table 2: Evaluation results for sentence-level and abstract-level claim verification across submissions.

Notably, the top-performing models benefited significantly from data augmentation, such as combining training and development sets, which yielded approximately a +5 F1 boost.

Discussion and Insights

Analysis of the submissions provided several critical insights for future research:

  1. The Necessity of Neural Refinement: Neural re-rankers are vital for bridging the gap between keyword-based retrieval and semantic relevance.
  2. Challenges in Dense Retrieval: Standard dense embedding approaches (e.g., BioSentVec) often underperform compared to well-tuned traditional methods like TF-IDF in specialized scientific domains.
  3. Contextual Importance: Providing the model with the full abstract context during evidence selection improves accuracy by 5.4% to 9.8%.
  4. Efficiency vs. Scale: While massive models like T5-3B are highly effective, optimized training techniques like aggressive negative sampling can allow much smaller models (e.g., RoBERTa-large) to achieve competitive performance.

Conclusion and Future Work

While the SciVer shared task demonstrated significant progress, several challenges remain. Current evaluations do not penalize computational complexity, potentially favoring oversized models that are difficult to deploy. Future work should focus on scaling to larger corpora, consolidating evidence from multiple documents, and developing efficient architectures that maintain high performance at a lower computational cost.

Summary updated

Abstract

We present an overview of the SciVer shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021. In this shared task, systems were provided a scientific claim and a corpus of research abstracts, and asked to identify which articles SUPPORT or REFUTE the claim as well as provide evidentiary sentences justifying those labels. 11 teams made a total of 14 submissions to the shared task leaderboard, leading to an improvement of more than +23 F1 on the primary task evaluation metric. In addition to surveying the participating systems, we provide several insights into modeling approaches to support continued progress and future research on the important and challenging task of scientific claim verification.
BibTeX
@inproceedings{wadden2021overview,
  author = {David Wadden and Kyle Lo},
  title = {{Overview and Insights from the SciVer Shared Task on Scientific Claim Verification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure