TL;DR
This paper addresses the critical challenge of evaluating text generation models by proposing new metrics that simultaneously measure both the quality and diversity of generated text. Existing metrics often focus on one aspect while neglecting the other, leading to incomplete assessments. The authors introduce MS-Jaccard (n-gram based) and Fréchet BERT Distance (feature-based) to provide a more holistic evaluation.
What Problem Does This Paper Address?
Evaluating the output of text generation models is notoriously difficult. While metrics like BLEU are widely used to assess the quality of generated sentences (e.g., how grammatically correct and semantically similar they are to reference texts), they fail to capture the diversity of the output. A model could consistently generate a single, high-quality sentence and achieve a good BLEU score, masking a lack of creativity or the problem of "mode collapse" where the generator gets stuck producing a limited set of outputs. Conversely, metrics like Self-BLEU attempt to measure diversity but ignore quality. This leaves a gap in comprehensively understanding a model's performance, especially for generative models like GANs which are prone to mode collapse. The authors aim to bridge this gap by developing metrics that capture both aspects simultaneously.
Methods
The paper proposes three novel metrics:
- MS-Jaccard: This metric uses n-grams to measure the similarity between the distribution of generated text and the distribution of real text. It's inspired by the Jaccard Index but operates on multi-sets of n-grams, considering their frequencies. The idea is that a model generating diverse and high-quality text will have an n-gram distribution that closely matches that of real data. Higher MS-Jaccard scores indicate better performance.
- Fréchet BERT Distance (FBD): Drawing inspiration from FID (Fréchet Inception Distance) used in image generation, FBD leverages the power of BERT embeddings. Text is passed through BERT to obtain feature representations, and then the Fréchet distance (also known as Wasserstein-2 divergence) is calculated between the Gaussian distributions of these BERT embeddings for real and generated text. This metric captures both quality and variety by considering the overall distribution of generated text in a rich feature space. Lower FBD scores are better.
- Oracle-Based Evaluation (Bhattacharyya distance): For scenarios where a "probabilistic oracle" (a model that can generate real data) is available, the authors propose using symmetric distances like the Bhattacharyya distance to measure the divergence between the oracle's distribution and the generative model's distribution. This is particularly useful for evaluating models trained in an "oracle training mode." Lower Bhattacharyya distances are better.
Datasets
The authors evaluate their proposed metrics on several standard datasets:
- COCO Captions: A large dataset of image captions.
- EMNLP2017 WMT News: A dataset for machine translation tasks.
- IMDB Movie Reviews: A dataset for sentiment analysis.
- Oracle synthetic dataset: A synthetic dataset generated by a randomly initialized LSTM, used for evaluating models in an oracle training setting.
Key Insights and Ideas
- The inadequacy of single-focus metrics: The paper strongly emphasizes that relying solely on quality (BLEU) or diversity (Self-BLEU) metrics provides an incomplete picture of text generation model performance.
- Leveraging BERT for text features: The use of BERT as a feature extractor for text generation evaluation is a key innovation, allowing for a more nuanced understanding of text semantics and structure beyond simple n-gram overlap.
- The importance of distribution matching: The proposed metrics are grounded in the idea of matching the distribution of generated text to the distribution of real text, whether through n-grams or feature embeddings.
- GANs struggle with quality-diversity balance: The results consistently show that while GAN-based models can be competitive, they often don't perform as well as simpler MLE-based models when evaluated on metrics that consider both quality and diversity. This reinforces the known challenge of mode collapse in GANs.
Key Results
The paper presents extensive experimental results across various datasets and models (MLE, SeqGAN, MaliGAN, RankGAN).
- Proposed metrics show promise: The proposed MS-Jaccard and FBD metrics generally align with the qualitative assessment of model performance, often highlighting the strengths of MLE models in balancing quality and diversity.
- MLE often outperforms GANs: In most evaluations, especially on real-world datasets, MLE models (using LSTM) tend to achieve better scores on the proposed joint quality-diversity metrics compared to GAN-based models. This suggests that GANs, despite their potential, still face challenges in generating text that is both high-quality and diverse.
- Correlation with existing analysis: The proposed metrics are shown to be correlated with existing methods of analyzing the quality-diversity spectrum (e.g., by varying temperature parameters), suggesting they can effectively capture model behavior across this spectrum.
- High correlation among proposed metrics: The paper demonstrates a high Pearson correlation between the proposed MS-Jaccard and FBD metrics on real-world datasets, indicating their consistency.
Limitations and Future Work
- Computational cost: While not explicitly detailed as a limitation, feature-based methods like FBD can be computationally more expensive than n-gram based metrics.
- Oracle availability: The oracle-based evaluation is contingent on having a probabilistic oracle, which might not always be feasible.
- Future work: The authors suggest that their proposed metrics can be further refined and applied to a wider range of text generation tasks. They also highlight the ongoing need for robust evaluation metrics in this rapidly evolving field.
Summary updated