Overview of the NTCIR-19 SciClaimEval Task: Cross-modal Claim Verification in Scientific Papers

Xanh Ho, André Greiner-Petter, Florian Boudin, Yun-Ang Wu, Sunisth Kumar, Tian Cheng Xia, Hiba Akbi, Tim Schopf, Akiko AizawaarXiv 2026 · arXiv

TL;DR

The NTCIR-19 SciClaimEval task introduces a new benchmark for verifying scientific claims against the tables and figures within the same paper. While top-performing multimodal large language models (MLLMs) achieved near-perfect scores by exploiting the dataset's contrastive structure, their performance plummeted when forced to evaluate claims in isolation—revealing that automated internal consistency checking in scientific literature remains a significant open challenge.

What Problem Does This Paper Address?

Scientific integrity relies on internal consistency: do the claims made in the text actually match the data presented in the tables and figures? During peer review, verifying these links is tedious and error-prone. While automated claim verification exists, most previous datasets focused on external fact-checking or relied on "refuted" examples created by simply flipping the wording of a claim (e.g., changing "increases" to "decreases").

Models often learn to spot these linguistic "surface cues" without actually looking at the evidence. SciClaimEval addresses this by keeping the claims authentic and instead modifying the evidence (the table or figure) to create a mismatch. This forces models to perform true cross-modal reasoning to determine if the data supports the text.

Methods

The organizers designed a pipeline to create a high-quality, challenging dataset and two distinct subtasks:

  1. Dataset Construction: They extracted claims and their corresponding evidence from peer-reviewed papers. For every "supported" claim-evidence pair, human annotators created a "refuted" version by minimally modifying the table or figure (e.g., changing a cell value or adding spurious data points to a plot).
  2. Subtask 1 (Claim-Label Prediction): Given a claim and one piece of evidence, the system must decide if the claim is "Supported" or "Refuted."
  3. Subtask 2 (Claim-Evidence Prediction): Given a claim and both versions of the evidence (original and modified), the system must pick the one that supports the claim.
Figure 1: Our dataset construction pipeline consists of three main steps: data preparation, automatic claim-evidence extraction, and human annotation. The human annotation process involves two tasks: claim-evidence verification and evidence modifica- tion. After collecting all samples, we design two subtasks in our dataset: claim-label prediction and claim-evidence prediction.

Participating teams primarily used Multimodal Large Language Models (MLLMs) like GPT-4o or Claude 3.5 Sonnet. Strategies ranged from direct prompting to complex pipelines involving claim decomposition, OCR for tables, and multi-agent debates.

Datasets

The authors introduced the SciClaimEval dataset, which consists of:

  • 1,633 claims derived from 180 papers.
  • Three domains: Natural Language Processing (ACL), Machine Learning (NeurIPS), and Medicine (PeerJ).
  • Multi-modal evidence: 987 tables and 646 figures.
  • Multiple formats: Tables were provided as images, LaTeX, HTML, and JSON to test how representation affects model performance.
Figure 3: Distribution of all figure types in test and dev split.

Key Insights and Ideas

  • The "Contrastive Shortcut": The most successful teams in Subtask 1 realized they could compare the two versions of the evidence provided in the test set. By looking for the difference between the two images/tables, they could easily deduce which one was "wrong."
  • The Performance Cliff: When this "paired" information was removed—forcing models to judge a single claim against a single image, as a human reviewer would—accuracy dropped by over 24 percentage points. This suggests current models are better at "spot the difference" than at deep scientific reasoning.
  • Open-Weight Parity: Interestingly, structured pipelines built around open-weight models (like Llama-3 or Qwen-VL) matched the performance of proprietary models like GPT-4o, provided they used clever prompting and verification steps.
  • Evidence Modification is Hard: Creating refuted versions of figures is significantly more complex than tables. Annotators had to manually flip graphs or swap legends to create logical but incorrect evidence.
Figure 2: An example from our dataset involves creating an unsupported claim by adding spurious data points to the figure. The context provides the necessary information to understand the plot in the evidence.

Key Results

  • Top Performance: The best systems (SciTrue and Bonn-Juelich Informatics) reached 98.1% pair accuracy on Subtask 1.
  • Unpaired Reality: The best system that did not exploit the paired structure (TakasuLab) scored only 73.8%, highlighting the difficulty of the task in a realistic setting.
  • Modality Matters: Models generally found tables easier to process than figures. On figures, performance varied wildly depending on the type of modification; "graph swaps" were particularly challenging for many teams.
Figure 5: Accuracy on Subtask 1 across figure modification categories. The red and green percentages indicate performance drops and gains, respectively, compared to the average accuracy across all figures (black triangle marker).

Limitations and Future Work

The authors acknowledge that the "paired" nature of the test set was a design flaw that allowed for shortcuts. Future iterations will ensure models cannot see both versions of the evidence simultaneously.

Additionally, the task currently uses a binary "Supported/Refuted" label, which doesn't capture the nuance of scientific writing (e.g., claims that are "partially supported" or "subjective"). Future work aims to expand the dataset to more complex domains like physics and materials science, incorporate citations as evidence, and move toward more nuanced, non-binary classification.

Summary updated

Abstract

This paper presents an overview of \emph{SciClaimEval}, a pilot task at NTCIR-19 on verifying scientific claims against the tables and figures of the papers in which they appear. Our dataset pairs authentic claims from 180 papers in machine learning, natural language processing, and medicine with contrastive versions of their tables and figures, one supporting the claim and one minimally modified so that it does not. The task comprises two subtasks: deciding whether a piece of evidence supports a claim, and selecting which of two evidence versions does. Twenty-two teams registered, the highest number among NTCIR-19 tasks, and 14 submitted 177 runs. Most systems built on multimodal large language models, relying on prompting and multi-stage inference rather than task-specific fine-tuning. On the first subtask, the best systems reached 98.1\% pair accuracy, but all six top-ranked systems relied on the contrastive pairs; the best system that judged each claim-evidence pair on its own scored 24.3 points lower. Open-weight models, embedded in carefully designed pipelines, matched proprietary ones. Yet verifying a single claim against its evidence, the setting that matters in practice, remains an open problem.
BibTeX
@misc{ho2026overview,
  author = {Xanh Ho and André Greiner-Petter and Florian Boudin and Yun-Ang Wu and Sunisth Kumar and Tian Cheng Xia and Hiba Akbi and Tim Schopf and Akiko Aizawa},
  title = {{Overview of the NTCIR-19 SciClaimEval Task: Cross-modal Claim Verification in Scientific Papers}},
  year = {2026},
  eprint = {2611.06090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06090},
}
Figure