SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables

Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, Iryna GurevychNeurIPS 2021 · OpenReview

TL;DR

This paper introduces SciGen, a novel dataset designed to push the boundaries of text generation from scientific tables. It challenges models to go beyond simply summarizing table content and instead requires them to perform arithmetic reasoning to generate accurate descriptions. While current state-of-the-art models struggle with this task, the dataset and proposed automatic extraction pipeline offer a path forward for developing more intelligent and capable text generation systems.

What Problem Does This Paper Address?

Generating natural language descriptions from structured data, known as data-to-text generation, is a well-established NLP task. However, existing datasets often focus on surface-level summarization. The authors identify a critical gap: describing scientific tables frequently necessitates arithmetic reasoning (e.g., comparisons, argMax, argMin) over numerical values. This is crucial for accurately conveying the findings presented in scientific literature. Current models, while adept at general text generation, often fail when faced with the need for such reasoning on complex inputs like tables, leading to factually incorrect or "hallucinated" descriptions. SciGen aims to address this by providing a benchmark specifically designed to evaluate and improve reasoning capabilities in data-to-text generation.

Methods

The core contribution of this paper is the creation of the SciGen dataset. To build this dataset, the authors employed a two-pronged approach:

  • Expert Annotation: A portion of the dataset was created through expert annotations by authors of the scientific articles. This involved identifying text spans that described specific tables and ensuring these descriptions could be generated from the table and its caption, specifically excluding information beyond the table's scope.
  • Automatic Extraction Pipeline: Recognizing that expert annotation is time-consuming and doesn't scale, the authors developed a pipeline to automatically extract high-quality table-description pairs from the LaTeX sources of scientific articles. This pipeline leverages the structure of LaTeX to locate table references and then applies post-processing heuristics to filter out irrelevant or low-quality pairs. This method allows for the creation of larger training sets, particularly for the "medium" and "large" splits of the dataset.

The paper also investigates the performance of state-of-the-art pre-trained language models (BART and T5) on this new dataset, using standard generation metrics and human evaluation.

Datasets

The primary contribution is the SciGen dataset, which consists of pairs of scientific tables and their corresponding natural language descriptions. SciGen is unique in that its tables primarily contain numerical values, and the descriptions require arithmetic reasoning. The dataset is released in three settings:

  • Few-shot: Contains only expert-annotated table-description pairs.
  • Medium: Includes few-shot data plus automatically extracted pairs from additional articles.
  • Large: Builds upon the medium setting with even more automatically extracted pairs.

The test set remains consistent across all settings, ensuring a fair evaluation. The authors also mention using existing datasets for evaluating the automatic extraction pipeline, comparing its output to expert annotations using metrics like BLEU and METEOR.

Key Insights and Ideas

  • Reasoning is Key: The most significant insight is that accurately describing scientific tables is not just about surface-level text generation; it fundamentally requires arithmetic reasoning. Models that lack this capability will inevitably produce incorrect or nonsensical outputs.
  • Automatic Extraction is Viable: The proposed pipeline demonstrates that it's possible to automatically extract valuable table-description pairs from scientific literature, significantly expanding the potential for training data beyond costly manual annotation.
  • Standard Metrics Fall Short: Traditional automatic evaluation metrics (BLEU, METEOR, etc.) are insufficient for assessing reasoning-aware text generation. They often fail to distinguish between factually correct and incorrect descriptions, highlighting the need for new evaluation paradigms.
  • Human Evaluation is Crucial: Human evaluation proved essential for understanding the true performance of models on SciGen, revealing limitations that automatic metrics missed.

Key Results

  • Model Performance: State-of-the-art models like BART and T5, despite being powerful, struggle significantly with SciGen. While their generated descriptions are often fluent and coherent, they are frequently factually incorrect due to a lack of arithmetic reasoning.
  • Impact of Automatic Data: Incorporating automatically extracted training data (in the medium and large settings) demonstrably improved the correctness of generated descriptions and reduced hallucinations compared to the few-shot setting.
  • Human Evaluation Findings: In human evaluations, only about 39% of generated statements were correct in the best-case scenarios. This indicates a substantial gap in models' ability to perform both reasoning and relevant content selection.
  • Metric Limitations: Automatic metrics showed very low scores and poor correlation with human judgments, underscoring their inadequacy for this task. For instance, incorrect descriptions sometimes received higher scores than correct ones on metrics like BLEU.

Limitations and Future Work

The authors acknowledge several limitations:

  • Automatic Data Quality: While beneficial, automatically extracted descriptions are sometimes longer than expert annotations and may contain information not directly derivable from the table and caption alone.
  • Gap in Reasoning: Even with improved training data, the current models still exhibit a significant gap in arithmetic reasoning capabilities.
  • Evaluation Metrics: The lack of reliable automatic metrics for reasoning-aware text generation is a major hurdle.

Future work should focus on developing better evaluation metrics that can capture factual correctness and reasoning. Further research into model architectures and training strategies that explicitly incorporate arithmetic reasoning is also crucial. The authors also suggest that their automatic extraction pipeline can be adapted to new domains, facilitating future research in reasoning-aware text generation across various fields.

Summary updated

Abstract

We introduce SciGen, a new challenge dataset consisting of tables from scientific articles and their corresponding descriptions, for the task of reasoning-aware data-to-text generation. Describing scientific tables goes beyond the surface realization of the table content and requires reasoning over table values. The unique properties of SciGen are that (1) tables mostly contain numerical values, and (2) the corresponding descriptions require arithmetic reasoning. SciGen is the first dataset that assesses the arithmetic reasoning capabilities of generation models on complex input structures, such as tables from scientific articles, and thus it opens new avenues for future research in reasoning-aware text generation and evaluation. The core part of SciGen, including the test data, is annotated by one of the authors of the corresponding articles. Such expert annotations do not scale to large training data sizes. To tackle this, we propose a pipeline for automatically extracting high-quality table-description pairs from the LaTeX sources of scientific articles. We study the effectiveness of state-of-the-art data-to-text generation models on SciGen and evaluate the results using common metrics and human evaluation. Our results and analyses show that adding high-quality unsupervised training data improves the correctness and reduces the hallucination in generated descriptions, however, the ability of state-of-the-art models is still severely limited on this task.
BibTeX
@inproceedings{moosavi2021scigen,
  author = {Nafise Sadat Moosavi and Andreas Rücklé and Dan Roth and Iryna Gurevych},
  title = {{SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=Jul-uX7EV_I},
}
Figure