Table-Text Alignment: Explaining Claim Verification Against Tables in Scientific Papers

Xanh Ho, Sunisth Kumar, Yun-Ang Wu, Florian Boudin, Atsuhiro Takasu, Akiko AizawaEMNLP 2025

TL;DR

This paper argues that simply predicting whether a scientific claim is supported or refuted by a table is insufficient. They introduce SciTabAlign, a new dataset that annotates the specific table cells needed for verification, and show that current LLMs, while good at predicting labels, struggle to identify these crucial explanatory cells, highlighting a gap in their reasoning.

What Problem Does This Paper Address?

Verifying claims against tabular data in scientific papers is a crucial task, but existing benchmarks often focus solely on predicting a "supported" or "refuted" label. This approach, the authors argue, offers little insight into why a model makes a certain prediction, hindering interpretability and trust. Scientific tables are also particularly challenging due to their density and domain-specific nature. While some datasets exist, they either use simplified, crowd-generated claims or lack explicit annotations for the evidence used in verification. This paper aims to bridge this gap by reframing claim verification as an explanation task, requiring models to not only predict a label but also identify the precise table cells that justify that prediction. This is vital for building more reliable and transparent AI systems for scientific literature analysis.

Methods

The core contribution of this work is the creation of the SciTabAlign dataset. This dataset extends the existing SciTab benchmark by introducing human-annotated cell-level rationales. For each claim-table pair, human annotators verified the claim's label and then highlighted the minimal set of table cells necessary to support their decision. This process involved two main tasks: claim label verification and evidence selection.

To ensure the quality and robustness of their dataset, the authors also developed a taxonomy of five common ambiguity types encountered during annotation. These include:

  • Table Conversion Errors: Artifacts from table extraction (e.g., merged cells, missing entries).
  • Additional Context Requirements: Claims needing information beyond the table itself (e.g., abbreviations, statistical tests).
  • Unexpected Claim Types: Descriptive or meta-level claims that don't require data reasoning.
  • Subjective Adjectives: Vague terms like "poor" or "substantial."
  • Unclear Claims: Ambiguous references to table elements.

The authors then used this dataset to evaluate various Large Language Models (LLMs), including table-specific models (TAPAS), open-source LLMs (Qwen 2.5, Llama 3.1), and closed-source LLMs (GPT-4o), employing different prompting strategies (zero-shot, few-shot, Chain-of-Thought). They also adopted the PIPE encoding format for tabular data representation, which has shown to be effective.

Datasets

  • SciTab: The base dataset used for extension. It contains naturally occurring claims against scientific tables.
  • SciTabAlign: The newly introduced dataset, which extends SciTab by adding human-annotated cell-level rationales for claim verification. The authors filtered out ambiguous cases and NEI (Not Enough Information) claims, resulting in 372 aligned samples (195 supported, 177 refuted).

Key Insights and Ideas

  • Beyond Label Prediction: The paper strongly advocates for moving beyond simple claim label prediction. The ability to identify supporting evidence is crucial for interpretability and building trust in AI systems.
  • LLMs Struggle with Explanation: Despite impressive performance on claim label prediction, even advanced LLMs like GPT-4o show significant limitations in identifying the correct, minimal set of table cells that justify their predictions. This suggests their reasoning might not be as "faithful" as their label predictions imply.
  • The Importance of "Minimal" Evidence: The focus on identifying the minimal set of cells is key. This forces models to pinpoint the most salient information, mirroring how a human expert would verify a claim.
  • Ambiguity is Pervasive: The proposed taxonomy of ambiguity types highlights the inherent challenges in scientific table-based verification, underscoring the need for more robust datasets and models that can handle these nuances.

Key Results

  • Claim Prediction: GPT-4o achieved the highest Macro-F1 score for claim label prediction. Larger models generally outperformed smaller ones. Prompting strategies like few-shot and CoT were beneficial, especially for smaller models.
  • Evidence Selection: This task proved significantly more challenging for all models. Even the best models fell short of human agreement scores (78.0 macro F1). GPT-4o, Llama 3.1 70B, and Qwen 2.5 72B showed comparable scores in zero-shot settings, with CoT prompting improving performance for Qwen 2.5 72B and Llama 3.1 70B. Smaller models struggled more with evidence selection.
  • Correlation: Analysis showed that while models often predict the correct claim label, they frequently fail to select the necessary supporting table cells, indicating a disconnect between accuracy and faithful reasoning.

Limitations and Future Work

  • Annotation Scale: The dataset is relatively modest in size (372 claims), which the authors acknowledge might affect statistical reliability and generalizability.
  • Domain Specificity: The dataset originates from the computer science domain, potentially limiting its applicability to tables and claims from other scientific fields.
  • Encoding Method: The PIPE encoding method might not be optimal for highly complex table structures, suggesting a need for more robust encoding approaches.

Future work could focus on expanding the dataset to include more domains and claims, developing more sophisticated table encoding methods, and designing models that can better bridge the gap between claim label prediction and faithful evidence selection.

Summary updated

BibTeX
@inproceedings{ho2025table,
  author = {Xanh Ho and Sunisth Kumar and Yun-Ang Wu and Florian Boudin and Atsuhiro Takasu and Akiko Aizawa},
  title = {{Table-Text Alignment: Explaining Claim Verification Against Tables in Scientific Papers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure