Harnessing CLIP for Evidence Identification in Scientific Literature: A Multimodal Approach to Context24 Shared Task

Anukriti Kumar, Lucy Lu Wang2024 · DOI

TL;DR

Researchers from the University of Washington have developed a multimodal AI approach to automatically link scientific claims to the figures and tables that support them. By fine-tuning the CLIP (Contrastive Language-Image Pre-training) model with enriched datasets and advanced text-image preprocessing, they secured third place in the Context24 Shared Task, demonstrating that data augmentation is key to bridging the gap between scholarly text and visual evidence.

What Problem Does This Paper Address?

In scientific literature, claims are rarely standalone; they rely on empirical evidence often tucked away in complex figures and tables. For researchers performing literature reviews or verifying facts, manually hunting for the specific chart or table that substantiates a claim is a time-consuming "needle in a haystack" problem.

While AI has made strides in text-based claim verification, multimodal evidence identification—linking a text claim to a visual asset—remains underexplored. The challenge lies in the "semantic gap": a claim might use high-level language, while a figure caption might be highly technical or specific to a single experiment. This paper tackles Task 1 of the Context24 Shared Task: predicting a ranked list of figures or tables from a paper that provide supporting evidence for a given claim.

Methods

The authors leveraged CLIP, a state-of-the-art multimodal model that learns to associate images and text in a shared mathematical space. Their approach involved three main pillars:

  1. Fine-tuning CLIP: They used the clip-vit-base-patch32 variant, training it to maximize the similarity between claims and their corresponding visual evidence.
  2. Data Enrichment with GPT-4: To provide the model with better context, they used GPT-4 to extract "sub-captions" (descriptions for specific parts of multi-panel figures) and "inline references" (the specific sentences in the paper's text that mention the figure).
  3. Advanced Preprocessing:
    • Text: They applied Named Entity Recognition (NER) and Semantic Role Labeling (SRL) to help the model understand the relationships between entities in the claims.
  4. Images: They used standard augmentations like random cropping, rotation, and color jittering to make the model more robust to different figure styles.

Datasets

The study utilized a combination of three primary sources to create a massive training pool:

  • Context24 Dataset: The core task dataset containing 474 scientific claims across biology, computer science, and social sciences.
  • SciMMIR: A large-scale collection of ~530,000 scientific image-text pairs extracted from arXiv.
  • MedICaT: A dataset of ~2,000 medical images with manually annotated sub-figures and sub-captions.
  • Augmented Training Data (ATD): A custom set created by the authors using GPT-4 to extract deeper context from the original Context24 papers.

Key Insights and Ideas

  • Context is King: Simply matching a claim to a figure caption isn't enough. By including "inline references" (how the author describes the figure in the body of the paper), the model gains a much better understanding of the figure's actual purpose.
  • The Power of Scale: Adding external datasets like SciMMIR and MedICaT led to a massive performance boost. NDCG@5 (a measure of ranking quality) jumped from 0.19 to 0.27 just by adding more data, even before any fancy preprocessing.
  • GPT-4 is a Double-Edged Sword: While GPT-4 was excellent at extracting sub-captions, it hallucinated or fabricated content in about 10% of cases. The authors had to implement a manual verification and heuristic filtering pipeline to ensure the "enriched" data didn't confuse the model.
  • Linguistic Structure Matters: Tagging named entities and roles (who did what to whom) in the text helped the model align the abstract language of a claim with the concrete data presented in a table or chart.

Key Results

The authors' best-performing model achieved:

  • NDCG@5: 0.26
  • NDCG@10: 0.30
  • Precision@5: 0.86 (with full text-image preprocessing on the validation split)

These results earned the team third place on the Context24 leaderboard. The experiments clearly showed that while image preprocessing helped, the combination of text-image preprocessing yielded the highest gains in precision and recall.

Limitations and Future Work

The authors identified several hurdles that still need to be cleared:

  • Complex Figures: Figures containing multiple sub-panels (e.g., Figure 1a, 1b, 1c) remain a major challenge, as captions often describe the whole set rather than individual components.
  • Domain Specificity: The current model uses broad datasets. The authors suggest that selecting subsets of data that more closely align with the specific scientific domain of the claim could further improve accuracy.
  • Claim Augmentation: Instead of just enriching the figure data, future work could involve using AI to expand the claim itself with more context before trying to match it to an image.

Summary updated

Abstract

Knowing whether scientific claims are supported by evidence is fundamental to scholarly communication and evidence-based decisionmaking.We present our approach to Task 1 of the Context24 Shared Task-Contextualizing Scientific Figures and Tables (SDP@ACL2024), which focuses on identifying multimodal evidence from scientific publications that support claims.We finetune CLIP, a state-of-the-art model for image-text similarity tasks, to identify and rank figures and tables in papers that substantiate specific claims.Our methods focus on text and image preprocessing techniques and augmenting the organizerprovided training data with labeled examples from the SciMMIR and MedICaT datasets.Our best-performing model achieved NDCG@5 and NDCG@10 values of 0.26 and 0.30, respectively, on the Context24 test split.Our findings underscore the effectiveness of data augmentation and preprocessing in improving the model's ability in evidence matching.
BibTeX
@misc{kumar2024harnessing,
  author = {Anukriti Kumar and Lucy Lu Wang},
  title = {{Harnessing CLIP for Evidence Identification in Scientific Literature: A Multimodal Approach to Context24 Shared Task}},
  year = {2024},
  doi = {10.18653/v1/2024.sdp-1.29},
}
Figure