TL;DR
This paper tackles the challenge of evaluating text generation models by proposing new metrics that simultaneously assess both the quality and diversity of generated text. By introducing measures like MS-Jaccard and Fréchet BERT Distance (FBD), the authors offer a more comprehensive evaluation framework than existing metrics that often focus on only one aspect, leading to a better understanding of model performance.
What Problem Does This Paper Address?
Evaluating the output of text generation models is a tricky business. While metrics like BLEU have been around for a while, they primarily focus on the quality of generated sentences, often by comparing them to reference texts. This can be misleading because a model might produce a few high-quality sentences but fail to generate a diverse range of outputs, a phenomenon known as "mode collapse." On the other hand, metrics like Self-BLEU try to capture diversity but completely ignore the quality of the generated text. This leaves a gap: how do we get a holistic view of a model's performance, considering both how good its outputs are and how varied they are? This paper aims to fill that gap by developing metrics that can jointly measure these two crucial aspects.
Methods
The authors propose three main categories of metrics:
- MS-Jaccard: This is an n-gram based measure that compares the distribution of n-grams in the generated text to that of real text. It's inspired by the Jaccard Index and aims to capture the similarity of the multisets of n-grams, considering their frequencies. The idea is that if a model generates text with similar n-gram patterns to real data, it's likely producing both quality and diverse outputs.
- Fréchet BERT Distance (FBD): Drawing inspiration from image generation evaluation (like FID), this metric uses BERT, a powerful pre-trained language model, as a feature extractor. It embeds real and generated text into a feature space and then calculates the Fréchet distance between the Gaussian distributions representing these embeddings. This approach captures both the quality and variety of sentences by considering their semantic representations.
- Oracle-Based Evaluation (Bhattacharyya): For scenarios where a "true" generative model (an oracle) can be defined, the authors propose using symmetric distances like Bhattacharyya to measure the divergence between the oracle's distribution and the model's distribution. This is particularly useful in "oracle training" modes where the generator's density can be estimated.
Datasets
The paper evaluates its proposed metrics on several standard datasets for text generation:
- COCO Captions: A dataset of image captions.
- EMNLP2017 WMT News: A dataset for machine translation tasks.
- IMDB Movie Reviews: A dataset for sentiment analysis.
Additionally, they use a synthetic oracle dataset generated by a randomly initialized LSTM, which is useful for evaluating metrics designed for oracle training scenarios.
Key Insights and Ideas
- The limitations of existing metrics are significant: BLEU and similar metrics can be gamed by models that produce a few excellent but repetitive sentences. Conversely, diversity metrics alone don't tell us if the generated text is actually coherent or useful.
- Quality and diversity are intertwined: The paper's core idea is that a good text generation model should excel at both. Their proposed metrics aim to reflect this by penalizing models that are strong in one aspect but weak in the other.
- Leveraging pre-trained models for evaluation: Using BERT as a feature extractor (FBD) is a smart way to tap into the rich semantic understanding of modern language models for evaluation, moving beyond simple n-gram overlap.
- The "temperature" parameter reveals quality-diversity trade-offs: The authors demonstrate how varying a "temperature" parameter in generative models affects the balance between quality and diversity, and their proposed metrics align with these observations.
Key Results
The paper evaluates several text generation models (MLE, SeqGAN, MaliGAN, RankGAN) using both existing and their proposed metrics across different datasets.
- MLE often performs best: On real-world datasets, the Maximum Likelihood Estimation (MLE) model, when evaluated with the proposed metrics, generally shows the best performance, indicating its ability to generate both quality and diverse text. This contrasts with GAN-based methods, which often struggle to achieve good scores on these joint metrics.
- Proposed metrics show consistency: The MS-Jaccard and FBD metrics are shown to be highly correlated with each other and also correlate well with NLL (a quality-focused metric), suggesting they provide a robust and consistent evaluation.
- Alignment with quality-diversity spectrum: The proposed metrics align with analyses that explore the quality-diversity trade-off by varying a temperature parameter, further validating their effectiveness. For instance, Figure 1 shows how models move across the quality-diversity space as temperature changes, and the proposed metrics track these movements.
Limitations and Future Work
- Computational cost: While not explicitly detailed as a limitation, calculating these new metrics, especially FBD which involves BERT embeddings, might be more computationally intensive than traditional metrics like BLEU.
- Further validation: While the paper shows strong correlations, further extensive evaluation across a wider range of models and tasks would strengthen the claims.
- Human evaluation: The paper acknowledges the difficulty of human evaluation for diversity, and while their metrics aim to automate this, direct comparison with human judgments on a large scale would be valuable.
- Exploring other distance measures: For the oracle-based evaluation, the paper mentions that a "wide range of distances" could be used, suggesting future work could explore other symmetric divergence measures.
Summary updated