A Tutorial on Evaluation Metrics used in Natural Language Generation

Mitesh M. Khapra, Ananya B. SaiNAACL 2021

TL;DR

The field of Natural Language Generation (NLG) is exploding, but evaluating its progress is a major challenge. This tutorial provides a comprehensive overview of existing and emerging automatic evaluation metrics for NLG, highlighting their strengths, weaknesses, and the ongoing research directions needed to better assess NLG systems.

What Problem Does This Paper Address?

The rapid advancement of Natural Language Generation (NLG) models, fueled by deep learning and vast datasets, has created a critical need for robust methods to track scientific progress. While human evaluation is the gold standard, it's prohibitively expensive and time-consuming. This has led to the widespread adoption of automatic evaluation metrics. However, many existing metrics, like BLEU, METEOR, and ROUGE, have faced significant criticism for their poor correlation with human judgments, their uninterpretability, and their bias towards specific models or tasks. This tutorial aims to address this gap by providing a structured overview of the evolution of NLG evaluation metrics, categorizing them, and discussing their shortcomings and future research avenues.

Methods

This tutorial is structured as a comprehensive overview and taxonomy of NLG evaluation metrics, rather than introducing a single novel method. The authors propose to cover:

  • Evolution of Metrics: Tracing the development from early metrics to current state-of-the-art.
  • Taxonomy of Metrics: Categorizing metrics into:
  • Context-free vs. Context-dependent: Whether they consider external information beyond the generated text.
  • Trained vs. Untrained (Heuristic-based): Whether they rely on learned models or predefined rules.
  • Task-specific vs. Task-agnostic: Whether they are designed for a particular NLG task or generalizable.
  • Detailed Analysis: Discussing specific types of metrics within these categories, including word-based, character-based, and embedding-based metrics, as well as feature-based and end-to-end trained metrics.
  • Criticisms and Shortcomings: Analyzing common issues such as poor correlations with human judgments, uninterpretability, bias, and limited adaptability.

Datasets

The paper does not introduce new datasets. Instead, it discusses the use of existing datasets implicitly through the evaluation of various NLG tasks like Machine Translation, Abstractive Summarization, Question Answering, Dialogue Systems, and Image Captioning. The references section lists numerous papers that have used various datasets for evaluating NLG models.

Key Insights and Ideas

  • The "Evaluation Gap": The core challenge is that while NLG models are rapidly improving, our ability to automatically and reliably measure that improvement is lagging behind.
  • Beyond BLEU: Traditional metrics like BLEU, while foundational, are increasingly insufficient due to their limitations in capturing the nuances of human language and their weak correlation with human perception of quality.
  • The Rise of Context: Newer metrics are moving towards incorporating context (e.g., source document, image, dialogue history) to provide more meaningful evaluations, leading to the distinction between "context-free" and "context-dependent" metrics.
  • Learned vs. Rule-Based: The field is seeing a split between traditional heuristic-based metrics and newer "trained" metrics that leverage machine learning to learn evaluation criteria.
  • Task-Specific Challenges: Different NLG tasks (e.g., translation vs. dialogue generation) present unique evaluation challenges, requiring tailored approaches.

Key Results

This is a tutorial paper, so it doesn't present experimental results in the traditional sense. Instead, it synthesizes existing research to provide a structured understanding of the field. The "key results" are the insights derived from analyzing the strengths and weaknesses of various metrics and the identification of research trends.

Limitations and Future Work

Limitations:

  • Uninterpretability: Many current metrics produce scores that are difficult to interpret, making it hard to understand why a system performs well or poorly.
  • Bias: Existing metrics can be biased towards certain types of models or generation styles.
  • Adaptability: Metrics often struggle to generalize across different NLG tasks.
  • Nuance Capture: The inability of many metrics to capture subtle aspects of text quality remains a significant challenge.

Future Work:

The authors suggest that future research should focus on developing metrics that:

  • Have better correlations with human judgments.
  • Are more interpretable.
  • Are less biased.
  • Can adapt to various NLG tasks.
  • Can better capture the nuances of generated text.
  • Explore the use of context more effectively.
  • Investigate the potential of end-to-end trained metrics.

Summary updated

BibTeX
@inproceedings{khapra2021tutorial,
  author = {Mitesh M. Khapra and Ananya B. Sai},
  title = {{A Tutorial on Evaluation Metrics used in Natural Language Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure