TL;DR
This paper evaluates several modern, pre-trained language model-based metrics (BERTScore, BLEURT, NUBIA, MoverScore, Mark-Evaluate) against traditional metrics like BLEU, using synthetically generated text with various impairments. The study reveals that none of the evaluated metrics consistently perform well across all types of semantic deterioration, with a notable lack of sensitivity to negation being a significant issue.
What Problem Does This Paper Address?
Evaluating the quality of text generated by Natural Language Generation (NLG) systems is a critical and increasingly difficult task. Traditional metrics like BLEU and ROUGE, based on n-gram overlap, have well-documented limitations and struggle with the increasingly sophisticated outputs of modern NLG models, which are often based on large pre-trained language models like BERT and GPT. To address this, newer "model-based" metrics have emerged, claiming higher correlation with human judgments. However, the robustness and true capabilities of these new metrics, especially when faced with common types of errors in generated text, are not fully understood. This paper aims to systematically investigate the sensitivity of these advanced metrics to various semantic and syntactic deteriorations, providing a comparative analysis against the established BLEU baseline.
Methods
The authors designed experiments to test the sensitivity of five model-based metrics (BERTScore, BLEURT, NUBIA, MoverScore, and Mark-Evaluate) and the baseline BLEU metric. They generated synthetic text by introducing specific types of impairments to original sentences from a news corpus. These impairments included:
- Word Swap: Randomly swapping word pairs to disrupt word order.
- Word Drop: Randomly removing words to simulate general quality degradation.
- Repetition: Repeatedly inserting a chosen sequence of words.
- Negation: Applying a syntactic change to negate the sentence's meaning.
- POS-Drop: Removing words based on their Part-of-Speech (POS) tags (e.g., determiners, nouns, verbs).
The metrics' scores were then analyzed to observe how they changed with increasing degrees of these impairments.
Datasets
The primary dataset used for generating the impaired texts was the CNN/Daily Mail data set. The authors selected this corpus because it consists of high-quality news articles, making the original sentences a suitable upper bound for evaluated metrics. No new benchmarks were introduced; instead, existing datasets were used to create controlled experimental conditions.
Key Insights and Ideas
- No Single Metric Reigns Supreme: The study highlights that no single model-based metric consistently outperforms others across all tested types of text deterioration. This suggests that a "one-size-fits-all" metric for NLG evaluation is still elusive.
- Negation is a Major Blind Spot: A critical finding is the widespread lack of sensitivity to negation among all evaluated metrics. This is particularly concerning as negation fundamentally alters sentence semantics.
- Varying Sensitivity to Word Order and Repetition: Model-based metrics exhibit diverse behaviors when faced with word order perturbations and repetition. Some metrics remain surprisingly insensitive, while others show a sharp decline.
- POS-Drop Reveals Nuances: The impact of dropping different POS tags varied. Notably, the removal of determiners, which are generally considered less semantically critical, did not significantly lower scores for most metrics compared to BLEU.
- Trained Metrics Show Peculiarities: BLEURT and NUBIA, which are trained models, sometimes showed unusual behavior, such as increasing scores with certain impairments or failing to decrease monotonically.
Key Results
- Negation: All metrics, including BLEU, showed a significant drop in scores for negated sentences, but none reported a "proper value," indicating a failure to accurately assess this severe semantic shift. NUBIA and BLEURT showed the lowest scores, averaging 0.65 and 0.77 respectively.
- Word Swap & Word Drop: BLEU, MoverScore, BERTScore, and Mark-Evaluate generally showed a monotonic decrease in scores as these impairments increased, often reaching near zero. However, BLEURT and NUBIA maintained higher scores, with NUBIA even increasing its score at higher impairment levels for word drop.
- Repetition: BERTScore metrics decreased monotonically to zero. BLEURT showed little sensitivity, converging to a high score (0.76), while NUBIA also increased its score.
- POS-Drop: The removal of determiners had minimal impact on most metrics. However, the impact of removing verbs, nouns, and adjectives varied across metrics. BERTScore consistently reported higher values than BLEU for POS-drop tasks.
Limitations and Future Work
The authors acknowledge a significant limitation: the lack of "expected ideal behavior" for metrics when confronted with modified samples. It's unclear how humans would rate these specific impaired cases, making it difficult to definitively label a metric's response as "correct" or "incorrect." This is especially true for negation, where the severity of the impairment and human perception are hard to quantify.
Future work should focus on:
- Improving Negation Sensitivity: Developing metrics that can accurately capture the semantic shift caused by negation is crucial.
- Ensembling Metrics: The authors endorse the idea of ensembling metrics, as different metrics show strengths in different areas. Combining them could lead to more robust evaluations.
- Further Investigation into Architecture Differences: Understanding why certain architectures (like those behind BLEURT and NUBIA) behave differently from others is needed.
- Human Evaluation: Incorporating human evaluation for these specific impairment tasks would provide a more grounded benchmark for metric performance.
- Addressing Sentence Complexity: The authors note that the long, complex sentences in the news corpus might pose challenges for some perturbation methods, suggesting this could be a factor in the results.
Summary updated