Abstract
Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the corresponding semi-structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PARENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall. Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the information extraction based evaluation proposed by Wiseman et al (2017), and show that PARENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using the data from the WebNLG challenge.
Introduction
The goal of table-to-text generation is to produce fluent, accurate natural language descriptions of structured data, such as Wikipedia infoboxes. Traditionally, progress in this field is measured using automatic metrics like BLEU and ROUGE, which compare model outputs to human-written "gold standard" references. However, this paper, presented at ACL 2019, identifies a fundamental flaw in this approach: many popular datasets are "noisy," containing reference texts that include information not found in the source tables.
The Problem of Divergent References
In the widely used WikiBio dataset, approximately 62% of human-written references contain facts (such as specific awards or middle names) that are absent from the associated infobox. The authors term these divergent references. When references diverge from the source data, standard metrics fail in two critical ways:
- Unfair Penalties: A model that is perfectly faithful to the table is penalized for not "hallucinating" the extra information found in the reference.
- False Rewards: A model that hallucinates incorrect information might receive a higher score if its errors happen to overlap with the reference's extra information.
This discrepancy makes it nearly impossible to use BLEU or ROUGE to reliably rank models or perform hyperparameter tuning, as the metrics do not account for the "anchor of truth" provided by the source table.
The PARENT Metric
To solve this, the authors propose PARENT (Precision And Recall of Entailed N-grams from the Table). Unlike text-only metrics, PARENT evaluates generated text against both the reference and the source table.
The core of PARENT is an entailment probability—an estimate of whether a specific n-gram in the generated text is supported by the table. This allows for two key improvements:
- Entailed Precision: A generated n-gram is considered correct if it appears in the reference OR if it is highly likely to be entailed by the table. This rewards models for being truthful to the data, even if the reference is incomplete.
- Entailed Recall: This combines the model's coverage of the reference (weighted by table support) with its coverage of the table itself.
By combining these into an F-score, PARENT balances faithfulness (accuracy) with coverage (completeness).
Experimental Evaluation
The researchers validated PARENT through a massive human evaluation study involving 8,800 pairwise comparisons across 16 different models. They tested the metric on two benchmarks: the highly divergent WikiBio and the cleaner WebNLG, where references were elicited specifically from RDF triples.
Key Results
PARENT demonstrated a significantly higher correlation with human judgment than existing metrics. On the WikiBio dataset, PARENT achieved a Pearson correlation of 0.91–0.98, while BLEU hovered around 0.55.
Furthermore, PARENT proved robust to varying levels of data divergence. In the "Hyperparams" experiment—which compares subtle variations of the same model—standard metrics like BLEU and BLEU-T showed unstable and even negative correlations with human judgment as the data became more divergent. In contrast, PARENT maintained a high, stable correlation.
Conclusion and Future Work
This research demonstrates that for data-to-text tasks, the source data is often a more reliable ground truth than heuristically collected references. While PARENT currently uses relatively simple lexical entailment, it sets a new standard for table-aware evaluation. Future work will explore integrating more sophisticated neural entailment models to handle complex paraphrasing and numerical reasoning, further closing the gap between automatic metrics and human understanding.
Summary updated