Not All Metrics Are Guilty: Improving NLG Evaluation with LLM Paraphrasing

Tianyi Tang, Hongyuan Lu, Yuchen Eleanor Jiang, Haoyang Huang, Dongdong Zhang, Wayne Xin Zhao, Furu Wei

TL;DR

This paper tackles the problem of unreliable Natural Language Generation (NLG) evaluation due to a lack of diverse reference sentences. The authors propose "Div-Ref," a method that uses Large Language Models (LLMs) to generate multiple paraphrased references for existing evaluation benchmarks. This simple yet effective approach significantly improves the correlation between automatic metrics and human judgments across various NLG tasks.

What Problem Does This Paper Address?

Evaluating the quality of generated text in NLG tasks like machine translation, summarization, and image captioning is crucial for progress. Current evaluation methods often rely on automatic metrics (e.g., BLEU, BERTScore) that compare generated text (hypotheses) against a limited set of human-written reference sentences. The core problem is that a single semantic meaning can be expressed in countless ways, and a few references simply cannot capture this full spectrum of valid expressions. This mismatch leads to poor correlations between automatic scores and human perception of quality, meaning a model might score well on a limited benchmark but still produce unsatisfactory output in real-world scenarios. The paper argues that the issue isn't necessarily with the metrics themselves, but with the impoverished evaluation benchmarks.

Methods

The paper introduces "Div-Ref," a method to enrich existing NLG evaluation benchmarks by increasing the number of reference sentences. The key innovation is leveraging the advanced paraphrasing capabilities of Large Language Models (LLMs) to generate diverse, semantically equivalent references from a single ground-truth reference.

Here's a breakdown of their approach:

  • Leveraging LLMs for Diversification: The authors use LLMs (specifically gpt-3.5-turbo-instruct) with carefully crafted prompts to generate multiple paraphrased versions of a given reference sentence. These prompts are designed to encourage variations in sentence structure, word choice, and even tone, while preserving the original meaning.
  • Prompt Engineering: They explore various prompt strategies, including simple paraphrasing and more complex instructions that guide the LLM to alter specific aspects of the sentence (e.g., order, structure, voice, tense, style).
  • Aggregation: For metrics that support multiple references (like BLEU), the scores are aggregated. The paper primarily uses maximum aggregation, suggesting that the highest score across the diversified references is most indicative of quality.
  • Compatibility: Div-Ref is designed to be orthogonal to existing automatic metrics. It can be applied to both traditional n-gram-based metrics and newer neural/LLM-based metrics, enhancing their performance.

Datasets

The paper evaluates their Div-Ref method on three established NLG evaluation benchmarks:

  • WMT22 Metrics Shared Task: This benchmark includes human scores for machine translation across multiple language pairs (Chinese-to-English, English-to-German, English-to-Russian).
  • SummEval: This dataset focuses on text summarization, with human judgments on coherence, consistency, fluency, and relevance.
  • PASCAL-50S: This dataset is used for image captioning evaluation, where human annotators compare captions against a reference.

The authors do not introduce a new benchmark but rather enhance existing ones by diversifying their reference sets.

Key Insights and Ideas

  • The "Reference Gap" is a Major Hurdle: The most significant insight is that the limited number of references in current benchmarks is a primary cause of poor metric-human correlation, not necessarily the metrics themselves.
  • LLMs are Powerful Paraphrasers for Evaluation: LLMs, with their sophisticated understanding of language and ability to follow instructions, are highly effective at generating diverse and semantically consistent paraphrases, making them ideal tools for this task.
  • Diversity is Key to Better Evaluation: By expanding the semantic coverage of references, Div-Ref helps automatic metrics better approximate human judgment, even for complex tasks and multilingual scenarios.
  • Simplicity and Agnosticism: The Div-Ref method is straightforward to implement and can be applied to a wide range of existing NLG metrics and tasks without requiring changes to the metric's core algorithms.
  • LLM-based Metrics Also Benefit: Even state-of-the-art LLM-based evaluation metrics see improvements when used with diversified references, highlighting the universal benefit of richer reference sets.

Key Results

The Div-Ref method consistently demonstrated significant improvements in the correlation between automatic evaluation metrics and human judgments across all tested benchmarks and metrics.

  • Machine Translation: Div-Ref improved correlations by an average of +7.82% across various metrics and language pairs on the WMT22 task. Notably, even the SOTA metric GEMBA saw enhancements.
  • Text Summarization: The method led to substantial improvements in correlation across all four evaluation aspects (coherence, consistency, fluency, relevance) for metrics like ROUGE, BERTScore, and the LLM-based ChatGPT-eval.
  • Image Captioning: Div-Ref showed enhancements in all metrics, particularly for SPICE, METEOR, and BERTScore, on the PASCAL-50S benchmark, even in challenging settings.
  • Robustness: The approach proved effective across different languages and metric types, including traditional word-based metrics and newer embedding-based and LLM-based metrics.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Computational Cost: Generating more references increases evaluation time. Future work could explore selection mechanisms to prioritize diverse sentences while minimizing the total number of references.
  • Domain Specificity: The current prompts might not perform optimally in specialized domains (e.g., finance, biomedicine) where professional terminology is critical. Developing domain-specific prompts is suggested.
  • LLM API Costs and Determinism: The reliance on LLM APIs like OpenAI's can be costly, and their non-deterministic nature might lead to slightly different results across runs. The potential for API models to be removed is also a concern.
  • Ablation Study Limitations: Due to cost, the ablation study for GEMBA was omitted, potentially leading to an incomplete analysis of LLM-based metrics.
  • Future Directions:
    • Investigating the optimal number of references for different metrics and tasks.
  • Exploring Div-Ref for NLG tasks in other modalities (e.g., speech, vision).
  • Examining whether paraphrasing can improve LLM training and utilization.

Summary updated

Abstract

Most research about natural language generation (NLG) relies on evaluation benchmarks with limited references for a sample, which may result in poor correlations with human judgements. The underlying reason is that one semantic meaning can actually be expressed in different forms, and the evaluation with a single or few references may not accurately reflect the quality of the model's hypotheses. To address this issue, this paper presents a novel method, named Para-Ref, to enhance existing evaluation benchmarks by enriching the number of references. We leverage large language models (LLMs) to paraphrase a single reference into multiple high-quality ones in diverse expressions. Experimental results on representative NLG tasks of machine translation, text summarization, and image caption demonstrate that our method can effectively improve the correlation with human evaluation for sixteen automatic evaluation metrics by +7.82% in ratio. We release the code and data at https://github.com/RUCAIBox/Para-Ref.
BibTeX
@inproceedings{tang2024not,
  author = {Tianyi Tang and Hongyuan Lu and Yuchen Eleanor Jiang and Haoyang Huang and Dongdong Zhang and Wayne Xin Zhao and Furu Wei},
  title = {{Not All Metrics Are Guilty: Improving NLG Evaluation with LLM Paraphrasing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=wvuaqIOC4z},
}
Figure