OSX at Context24: How Well Can GPT Tackle Contexualizing Scientific Figures and Tables

Tosho HirasawaACL 2024

TL;DR

This paper explores how well large language models (LLMs) like GPT-4 can automatically identify supporting figures/tables and methodological details for scientific claims, a crucial task for understanding research papers. The authors developed zero-shot GPT-4 based systems that achieved high performance on the Context24 shared task, demonstrating LLMs' potential for scientific document processing without extensive task-specific training.

What Problem Does This Paper Address?

Scientific papers are complex, and understanding the relationship between textual claims and supporting visual elements (figures, tables) and methodological details is fundamental for effective communication and knowledge dissemination. The Context24 shared task aims to automate this process by identifying key figures/tables that support a claim and relevant methodological details associated with it. This is particularly challenging in low-resource domains where large annotated datasets for supervised learning are scarce.

Methods

The authors propose zero-shot systems leveraging GPT-4 models for both subtasks of the Context24 shared task:

  • Evidence Identification: This task involves identifying key figures or tables that support a given scientific claim. The system uses a pipeline approach where GPT-40 calculates a "supportiveness score" for each claim-image pair. This score is based on comparing the claim with the image and its caption. To improve accuracy, the prompt was designed to penalize images that contain excessive information not directly relevant to the claim.
  • Grounding Context Identification: This task focuses on identifying methodological details related to a scientific claim. The system uses GPT-4-turbo, which can handle longer contexts, to extract relevant text snippets from the full research paper. The prompt guides GPT-4-turbo to search for specific aspects of empirical methods, such as observable measures, methods/analyses, and participants/datasets.

Datasets

The paper utilized the official dataset provided for the Context24 shared task for validation and testing. For evaluation, they created a balanced validation dataset by randomly selecting 10 instances from each of the four dataset categories: BIOL403, akamatsulab, dg-social-media-polarization, and megacoglab.

Key Insights and Ideas

  • LLMs as powerful zero-shot learners for scientific document processing: The core insight is that LLMs like GPT-4 can perform complex tasks like identifying supporting evidence and methodological details without requiring task-specific fine-tuning or large annotated datasets. This is a significant advantage for low-resource domains.
  • Prompt engineering is crucial for LLM performance: The authors found that carefully crafted prompts, including instructions to articulate reasoning and penalize irrelevant information, significantly improve the LLM's performance.
  • CLIP's limitations in scientific image understanding: The CLIP-based baseline struggled with identifying key scientific images, suggesting that its ability to distinguish subtle differences in scientific visuals is limited compared to LLMs. This highlights the unique challenges of multimodal scientific understanding.
  • GPT-4's ability to handle cross-modal and unimodal information: The success of the GPT-4 systems indicates its capability to integrate information from both visual (figures/tables) and textual modalities, as well as purely textual information, to support scientific claims.

Key Results

  • The proposed GPT-4-based systems achieved the second-highest rank in both the Evidence Identification and Grounding Context Identification tracks of the Context24 shared task.
  • For Evidence Identification, the GPT-4o system achieved a NDCG@5 score of 0.63 on the test data, more than double the random baseline.
  • For Grounding Context Identification, the GPT-4-turbo system achieved a BERTScore of 0.86 and ROUGE-L of 0.26 on the test data, performing comparably to the first-placed model with minimal prompt tuning.
  • The paper also noted that a significant portion of images were flagged as "prohibited" by GPT-40, suggesting potential issues with safety filters or prompt interpretation that could be addressed by rephrasing prompts.

Limitations and Future Work

  • Prohibited Images: A small percentage of images were flagged as prohibited by GPT-40. The authors suggest that rephrasing prompts or clarifying the purpose of the analysis could mitigate this.
  • Dataset Distribution Differences: The performance of the Evidence Identification system varied across datasets, with better performance on BIOL403 and dg-social-media-polarization and worse performance on akamatsulab and megacoglab. This suggests that the model's performance is sensitive to the distribution of the data.
  • Fine-tuning CLIP limitations: While fine-tuning CLIP was explored, the NLI-based approach underperformed the GPT-40 pipeline, indicating that images from the same article might be more relevant than those from different articles.
  • Future Work: The authors implicitly suggest further prompt engineering and exploration of LLM capabilities for scientific document understanding. The success of zero-shot approaches also opens avenues for applying these methods to other low-resource scientific domains.

Summary updated

BibTeX
@inproceedings{hirasawa2024osx,
  author = {Tosho Hirasawa},
  title = {{OSX at Context24: How Well Can GPT Tackle Contexualizing Scientific Figures and Tables}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure