Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo

Barkavi Sundararajan, Somayajulu Sripada, Ehud ReiterNAACL 2024 · DOI

TL;DR

Neural table-to-text models often "hallucinate" factual errors. This paper investigates how input data quality contributes to these errors and demonstrates that by systematically correcting problematic tabular inputs, factual accuracy can be significantly improved, reducing errors by up to 76%.

What Problem Does This Paper Address?

Generating natural language descriptions from tables (table-to-text) is a crucial task with applications ranging from medical reports to sports summaries. However, neural language models, despite their fluency, frequently produce outputs that are factually incorrect or unrelated to the input table – a phenomenon known as "hallucination." This paper focuses on a specific dataset, ToTTo, and investigates whether these hallucinations can be traced back to issues within the tabular input itself, rather than solely being a limitation of the generation models. The authors identify specific types of problematic tabular structures that lead to these factual errors.

Methods

The core of this research involves a meticulous manual error analysis of generated texts and their corresponding tabular inputs. The authors first identify factual errors in model outputs and then trace these errors back to specific problems in the input tables. They then propose and implement manual "fixes" for these identified input problems. These corrected inputs are then fed back into the models to evaluate the impact on factual accuracy. The study employs a systematic error annotation methodology, categorizing errors and analyzing their root causes in the input data. They also explore the effectiveness of these corrections on different model architectures, including T5 and LLAMA 2.

Datasets

The primary dataset used in this paper is ToTTo (Table-to-Text Generation). The authors specifically focus on the "politics" domain of this dataset for their analysis. They also mention using a subset of 210 samples for T5 models and 40 challenging samples for LLAMA 2 models, derived from the error samples identified in their pilot study. No new benchmarks were introduced; the focus was on analyzing and correcting existing data within ToTTo.

Key Insights and Ideas

  • Input Quality is Key: The research strongly suggests that a significant portion of factual errors in table-to-text generation stems from problematic input data structures, not just model limitations.
  • "Non-Standard" Tables are Trouble: Models struggle with tables that have non-atomic cell values (multiple pieces of information in one cell), missing crucial information, or complex/nested headers.
  • Fixing Inputs, Fixing Outputs: Systematically correcting these input issues leads to a substantial reduction in factual errors, demonstrating the effectiveness of data pre-processing.
  • Model Agnosticism (to an extent): While different models show varying degrees of improvement, the core insight about input problems and their correction holds across the tested architectures.
  • Manual Annotation is Crucial: Automatic metrics like BLEU and ROUGE are insufficient for pinpointing specific error types; manual error analysis is essential for understanding and addressing the root causes.

Key Results

The paper reports significant improvements in factual accuracy after correcting input problems:

  • T5 Models: Factual errors were reduced by 52% to 76% for LLAMA 2 models and 57% to 62% for T5 models when using corrected inputs.
  • LLAMA 2 Models: Error reductions of 52% (LLAMA 2-7B) and 76% (LLAMA 2-13B) were observed on challenging samples after input correction.
  • Error Categories: Input correction effectively reduced errors related to non-atomic cell values, complex table types, and insufficient input.
  • Omissions Increase: Interestingly, while factual errors decreased, the rate of "omissions" (where the model omits information) sometimes increased after input correction, particularly with complex tables or when multiple records were involved.

Limitations and Future Work

  • Domain Specificity: The study was limited to the "politics" domain of the ToTTo dataset. Generalizing these findings to other domains might require further investigation.
  • Scope of Corrections: The corrections were manual and focused on specific identified input problems. Extending this to the entire ToTTo dataset, especially the training split, was deemed too time-consuming.
  • Model Generalization: The experiments were conducted on T5 and LLAMA 2 models, and the results might not directly generalize to all other table-to-text models.
  • Future Work: The authors plan to investigate why omissions increase after input correction. They also aim to explore other tabular datasets and study the generalization of their proposed fixes. Simplifying the definitions of "context" and "addition" error categories is also a future goal, as annotation revealed some disagreements.

Summary updated

BibTeX
@inproceedings{sundararajan2024improving,
  author = {Barkavi Sundararajan and Somayajulu Sripada and Ehud Reiter},
  title = {{Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.408},
}
Figure