Overview of the Context24 Shared Task on Contextualizing Scientific Claims

Chu Sern Joel Chan, Aakanksha Naik, Matthew Akamatsu, Hanna Bekele, Erin Bransom, Ian Campbell, Jenna SparksACL 2024

TL;DR

This paper introduces Context24, a new dataset and shared task designed to help machines understand scientific claims by linking them to supporting figures, tables, and methodological details. This is crucial for accurate scientific interpretation, especially when claims are encountered outside their original context, and the shared task results show promising progress in this challenging area.

What Problem Does This Paper Address?

Interpreting scientific claims accurately, especially when they're presented in isolation (e.g., in literature reviews, presentations, or online discussions), is a significant challenge. Simply knowing a claim was asserted isn't enough; understanding its validity requires contextualizing it with the empirical evidence (like figures and tables) and methodological details (how the study was conducted) that support it. Humans find this retrieval process time-consuming and difficult. While existing NLP models can verify claims against text or generate figure captions, there's a lack of resources specifically for grounding scientific claims with their underlying empirical and methodological evidence. This gap can lead to the spread of "ghost claims" that lack solid evidence.

Methods

The paper describes the creation of the Context24 dataset and the organization of a shared task around it. The task has two main tracks:

  • Track 1: Evidence Identification: This track models the problem as an image-text retrieval task. Given a scientific claim and a paper's figures and tables, models must rank these visual elements based on how well they support the claim.
  • Track 2: Grounding Context Identification: This track frames the problem as query-based extractive summarization. Given a scientific claim and the full text of a research paper, models must extract text snippets that describe the key methodological details supporting the claim.

The authors also discuss various baseline approaches, including text-based retrieval using embeddings (MPNet, SPECTER), LLM prompting (GPT-4, GPT-4 Turbo, Mixtral-8x22B), and multimodal LLMs (LLaVA).

Datasets

The core contribution is the Context24 dataset, comprising 585 scientific claims.

  • Track 1: Includes gold annotations for figures/tables that ground the empirical evidence for each claim.
  • Track 2: Includes gold text snippets of methodological details for a subset of claims.

The dataset was curated from naturalistic synthesis work across various research contexts (cell biology, HCI, social sciences, etc.). The paper also references existing datasets for claim verification (SciFact, COVID-Fact), scientific figure captioning (SciCap, MedICaT), and data extraction (EBM-NLP, PubMed PICO), highlighting how Context24 extends these by focusing on empirical evidence and methodological details across broader scientific domains.

Key Insights and Ideas

  • Empirical vs. Textual Evidence: The paper strongly emphasizes the distinction between textual evidence (a claim was stated) and empirical evidence (figures/tables supporting the claim). This distinction is critical for robust sensemaking and avoiding the propagation of unsubstantiated claims.
  • Compound Figures: A significant challenge identified is that figures are often compound (containing multiple sub-figures), and only a subset might support a claim. This complicates evaluation and model design, requiring nuanced relevance scoring.
  • LLMs Show Promise: Both text-based and multimodal LLMs demonstrated strong performance, particularly in Track 1, with some models approaching or exceeding human-level performance on certain metrics. This suggests LLMs are becoming increasingly capable of understanding and contextualizing scientific information.
  • PDF Preprocessing is a Bottleneck: The authors highlight that current PDF processing tools struggle with accurate extraction of figures and tables, especially for complex domains like biology. This necessitates manual extraction for dataset creation and points to a crucial area for future research.

Key Results

  • Track 1 Performance: The top-performing team (CSIRO-LT) achieved nDCG@5 = 0.73 and nDCG@10 = 1.06. Baseline models like GPT-4o showed surprisingly strong zero-shot performance, indicating the growing tractability of the task for closed LLMs.
  • Track 2 Performance: Participant teams (CSIRO-LT and OSX) achieved close scores, moderately improving over baselines. BERTScore was generally higher than ROUGE scores, suggesting that while overall text similarity is captured, fine-grained factual accuracy might be a challenge. Mixtral-8x22B showed competitive performance with GPT-4, indicating open LLMs are also viable.
  • Difficulty of the Task: The low performance of many baseline systems and the fact that even top teams had room for improvement underscore the inherent difficulty of accurately contextualizing scientific claims.

Limitations and Future Work

  • PDF Preprocessing: The paper acknowledges the limitations of current PDF parsing tools and suggests that improving end-to-end PDF processing for figure/table extraction is essential for future systems.
  • Evaluation Metrics: The authors propose that current metrics for context identification (Track 2) might not fully capture the nuances of methodological detail extraction. They suggest exploring factuality-based evaluations or LLM-based judges for more robust assessment.
  • Image-Text Similarity Models: While multimodal LLMs performed well, the paper notes that contrastive models like CLIP didn't perform as strongly as expected. Further research into why these models struggle and how to adapt them for scientific domains is recommended.
  • Dataset Expansion: The dataset is a valuable resource, but future work could involve expanding it to cover more scientific domains and potentially incorporating other forms of evidence.

Summary updated

BibTeX
@inproceedings{chan2024overview,
  author = {Chu Sern Joel Chan and Aakanksha Naik and Matthew Akamatsu and Hanna Bekele and Erin Bransom and Ian Campbell and Jenna Sparks},
  title = {{Overview of the Context24 Shared Task on Contextualizing Scientific Claims}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure