TL;DR
The NTCIR-19 SciClaimEval task introduces a new benchmark for verifying scientific claims against the tables and figures within the same paper. While top-performing multimodal large language models (MLLMs) achieved near-perfect scores by exploiting the dataset's contrastive structure, their performance plummeted when forced to evaluate claims in isolation—revealing that automated internal consistency checking in scientific literature remains a significant open challenge.
What Problem Does This Paper Address?
Scientific integrity relies on internal consistency: do the claims made in the text actually match the data presented in the tables and figures? During peer review, verifying these links is tedious and error-prone. While automated claim verification exists, most previous datasets focused on external fact-checking or relied on "refuted" examples created by simply flipping the wording of a claim (e.g., changing "increases" to "decreases").
Models often learn to spot these linguistic "surface cues" without actually looking at the evidence. SciClaimEval addresses this by keeping the claims authentic and instead modifying the evidence (the table or figure) to create a mismatch. This forces models to perform true cross-modal reasoning to determine if the data supports the text.
Methods
The organizers designed a pipeline to create a high-quality, challenging dataset and two distinct subtasks:
- Dataset Construction: They extracted claims and their corresponding evidence from peer-reviewed papers. For every "supported" claim-evidence pair, human annotators created a "refuted" version by minimally modifying the table or figure (e.g., changing a cell value or adding spurious data points to a plot).
- Subtask 1 (Claim-Label Prediction): Given a claim and one piece of evidence, the system must decide if the claim is "Supported" or "Refuted."
- Subtask 2 (Claim-Evidence Prediction): Given a claim and both versions of the evidence (original and modified), the system must pick the one that supports the claim.
Participating teams primarily used Multimodal Large Language Models (MLLMs) like GPT-4o or Claude 3.5 Sonnet. Strategies ranged from direct prompting to complex pipelines involving claim decomposition, OCR for tables, and multi-agent debates.
Datasets
The authors introduced the SciClaimEval dataset, which consists of:
- 1,633 claims derived from 180 papers.
- Three domains: Natural Language Processing (ACL), Machine Learning (NeurIPS), and Medicine (PeerJ).
- Multi-modal evidence: 987 tables and 646 figures.
- Multiple formats: Tables were provided as images, LaTeX, HTML, and JSON to test how representation affects model performance.
Key Insights and Ideas
- The "Contrastive Shortcut": The most successful teams in Subtask 1 realized they could compare the two versions of the evidence provided in the test set. By looking for the difference between the two images/tables, they could easily deduce which one was "wrong."
- The Performance Cliff: When this "paired" information was removed—forcing models to judge a single claim against a single image, as a human reviewer would—accuracy dropped by over 24 percentage points. This suggests current models are better at "spot the difference" than at deep scientific reasoning.
- Open-Weight Parity: Interestingly, structured pipelines built around open-weight models (like Llama-3 or Qwen-VL) matched the performance of proprietary models like GPT-4o, provided they used clever prompting and verification steps.
- Evidence Modification is Hard: Creating refuted versions of figures is significantly more complex than tables. Annotators had to manually flip graphs or swap legends to create logical but incorrect evidence.
Key Results
- Top Performance: The best systems (SciTrue and Bonn-Juelich Informatics) reached 98.1% pair accuracy on Subtask 1.
- Unpaired Reality: The best system that did not exploit the paired structure (TakasuLab) scored only 73.8%, highlighting the difficulty of the task in a realistic setting.
- Modality Matters: Models generally found tables easier to process than figures. On figures, performance varied wildly depending on the type of modification; "graph swaps" were particularly challenging for many teams.
Limitations and Future Work
The authors acknowledge that the "paired" nature of the test set was a design flaw that allowed for shortcuts. Future iterations will ensure models cannot see both versions of the evidence simultaneously.
Additionally, the task currently uses a binary "Supported/Refuted" label, which doesn't capture the nuance of scientific writing (e.g., claims that are "partially supported" or "subjective"). Future work aims to expand the dataset to more complex domains like physics and materials science, incorporate citations as evidence, and move toward more nuanced, non-binary classification.
Summary updated