Abstract
While powerful pre-trained language models have improved the fluency of text generation models, semantic adequacy—the ability to generate text that is semantically faithful to the input—remains an unsolved issue. In this paper, we introduce a novel automatic evaluation metric, Entity-Based Semantic Adequacy (ESA), which assesses the extent to which generation models verbalizing RDF (Resource Description Framework) graphs produce text containing mentions of the entities occurring in the input. We compare 25 models from the WebNLG Shared Tasks and examine correlations with human evaluations. Our results suggest that ESA provides a more reliable, objective measure of input verbalization than subjective human measures.
The Challenge: Fluency vs. Accuracy
In Data-to-Text (D2T) generation, the goal is to transform structured data triples—such as those found in RDF graphs—into natural, human-readable sentences. While modern Large Language Models (LLMs) excel at producing fluent, grammatical text, they often struggle with semantic adequacy.
Models frequently suffer from two primary failure modes:
- Omissions: Failing to mention entities present in the source data.
- Hallucinations: Inventing facts or entities that were never in the input.
Standard metrics like BLEU or METEOR measure surface-level similarity to a reference text but often fail to penalize a model for ignoring critical input data or hallucinating new information.
The ESA Metric
To address these gaps, we propose ESA, a metric that calculates the ratio of entities from the input graph successfully detected in the generated output. The metric relies on a robust multi-stage detection pipeline:
- Entity Linking: Mapping text segments to DBpedia entities using state-of-the-art linkers.
- Approximate String Matching: Using fuzzy matching (Levenshtein distance) to account for naming variations (e.g., "German Empire" vs. "The Empire of Germany").
- Pronominal Coreference: Resolving pronouns like "she" or "it" back to their source entities.
- Date Normalization: Aligning disparate date formats to ensure temporal accuracy.
Experimental Setup and Insights
We evaluated ESA using the WebNLG Shared Tasks (2017 and 2020), analyzing a corpus of over 25,000 (RDF, Text) pairs across 25 different models.
Fluency as a Mask
A critical finding of our study is that high BLEU scores can be deceptive. As shown in the comparison of model rankings, several systems that achieved top-tier fluency scores performed poorly on semantic adequacy. This confirms that a "well-written" sentence is not necessarily a faithful one.
The "17% Gap"
Even among advanced 2020 models, the average system failed to mention at least one required entity in 17% of its outputs. While this is an improvement over 2017 models (where failure rates ranged from 10% to 77%), it highlights that semantic adequacy remains an unsolved hurdle for reliable NLG.
Key Results
- Reliability: Our entity detection algorithm achieved a recall of 0.74 and precision of 0.75, making it a strong proxy for manual annotation.
- Human Correlation: ESA shows a strong correlation with human judgments of "Semantics" and "Data Coverage," significantly outperforming BLEU in predicting human adequacy ratings.
- Objectivity: We found that human evaluators are often inconsistent, sometimes rewarding fluent but inaccurate text. ESA provides a more objective, reproducible baseline for data coverage.
Limitations and Future Work
Currently, ESA focuses on entities (the subjects and objects) rather than predicates (the relationships). A model could correctly identify "Paris" and "France" but incorrectly describe their relationship. Future work will involve extending this framework to include relation-based adequacy, ensuring the logical links in the generated text match the input graph structure.
Summary updated