On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation

Tiago Pimentel, Clara Meister, Ryan Cotterell

Abstract

A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality. Yet, there is a dearth of such metrics, which inhibits the rapid and efficient progress of language generators. One exception is the recently proposed Mauve. In theory, Mauve measures an information-theoretic divergence between two probability distributions over strings: one representing the language generator under evaluation; the other representing the true natural language distribution. Mauve's authors argue that its success comes from the qualitative properties of their proposed divergence. Yet in practice, as this divergence is uncomputable, Mauve approximates it by measuring the divergence between multinomial distributions over clusters instead, where cluster assignments are attained by grouping strings based on a pre-trained language model's embeddings. As we show, however, this is not a tight approximation -- in either theory or practice. This begs the question: why does Mauve work so well? In this work, we show that Mauve was right for the wrong reasons, and that its newly proposed divergence is not necessary for its high performance. In fact, classical divergences paired with its proposed cluster-based approximation may actually serve as better evaluation metrics. We finish the paper with a probing analysis; this analysis leads us to conclude that -- by encoding syntactic- and coherence-level features of text, while ignoring surface-level features -- such cluster-based substitutes to string distributions may simply be better for evaluating state-of-the-art language generators.

Introduction

Evaluating the output of open-ended text generators (like GPT-4) is a significant challenge in NLP. While human evaluation remains the gold standard, its cost and latency necessitate reliable automatic metrics. Traditional metrics like BLEU or ROUGE, which rely on surface-level n-gram overlap, often fail to capture the semantic and logical nuances of high-quality generation.

Recently, MAUVE has emerged as a state-of-the-art metric due to its high correlation with human judgment. Theoretically, MAUVE calculates the Area Under the Curve (AUC) of a specific information-theoretic divergence between human and machine text distributions. However, to make this computable, MAUVE maps infinite string distributions into finite clusters based on model embeddings. This research investigates whether MAUVE's success stems from its complex mathematical divergence or simply from this cluster-based representation.

Methodology

The study deconstructs MAUVE by comparing it against classical statistical divergences, including Forward and Backward KL Divergence and Jensen-Shannon (JS) Divergence. The authors evaluate these metrics in two settings:

  1. String-based: Using n-gram models to represent text distributions.
  2. Cluster-based: Using multinomial distributions over clusters derived from GPT-2 or BERT embeddings.

To understand the underlying mechanics, the team performed probing analyses (predicting sentiment, topic, and authorship from clusters) and perturbation experiments (e.g., shuffling words or removing articles) to see which linguistic features the clusters actually capture.

The Cluster Approximation Paradox

A central finding of the paper is that the cluster-based method used in MAUVE is not a faithful approximation of the theoretical string-based divergence. In fact, the authors discovered that cluster-based probabilities often correlate negatively with true string probabilities.

Correlation between cluster-based (q_c) and string-based (q_w) probabilities across model sizes.

As shown in the figure above, while word-based probabilities ($\hat{q}_w$) show high positive correlation, the cluster-based probabilities ($\hat{q}_c$) exhibit a negative correlation. This suggests that MAUVE succeeds not because it approximates the intended string divergence, but because the clusters themselves provide a more useful representation for evaluation.

Key Results

The experiments demonstrate that clusters are the "secret sauce" of MAUVE. When classical divergences like JS or KL are applied to embedding clusters, they perform as well as, or better than, MAUVE’s specific AUC metric.

Correlation of various cluster-based and string-based metrics with human judgments of Interestingness, Sensibility, and Human-likeness.

Key findings from the results include:

  • Superior Correlation: Cluster-based versions of all analyzed divergences correlate more strongly with human judgments of "interestingness" and "sensibility" than string-based versions.
  • JS Divergence Performance: On the "Human-likeness" task, Jensen-Shannon (JS) divergence on clusters achieved a correlation of ~95%, slightly outperforming MAUVE’s ~92%.
  • Stability: Cluster-based metrics showed significantly lower variance across different random seeds, offering higher reliability for researchers.

Discussion and Conclusion

Probing reveals that embedding clusters are highly sensitive to "discourse-level" features like sentiment and coherence but are largely "blind" to surface-level errors like missing articles. This "blindness" is actually a feature: because modern LLMs are already proficient at basic grammar, metrics need to ignore surface fluency to distinguish between high-level logic and coherence.

While highly effective, these metrics could potentially be "gamed" by models that produce logically coherent but grammatically broken text. Future work should focus on developing discretization methods that align with specific linguistic attributes rather than relying solely on the "black box" of pre-trained embeddings. In summary, MAUVE was "right for the wrong reasons"—its success validates the power of embedding clusters over its specific information-theoretic formulation.

Summary updated

Abstract

A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality. Yet, there is a dearth of such metrics, which inhibits the rapid and efficient progress of language generators. One exception is the recently proposed Mauve. In theory, Mauve measures an information-theoretic divergence between two probability distributions over strings: one representing the language generator under evaluation; the other representing the true natural language distribution. Mauve's authors argue that its success comes from the qualitative properties of their proposed divergence. Yet in practice, as this divergence is uncomputable, Mauve approximates it by measuring the divergence between multinomial distributions over clusters instead, where cluster assignments are attained by grouping strings based on a pre-trained language model's embeddings. As we show, however, this is not a tight approximation -- in either theory or practice. This begs the question: why does Mauve work so well? In this work, we show that Mauve was right for the wrong reasons, and that its newly proposed divergence is not necessary for its high performance. In fact, classical divergences paired with its proposed cluster-based approximation may actually serve as better evaluation metrics. We finish the paper with a probing analysis; this analysis leads us to conclude that -- by encoding syntactic- and coherence-level features of text, while ignoring surface-level features -- such cluster-based substitutes to string distributions may simply be better for evaluating state-of-the-art language generators.
BibTeX
@misc{pimentel2022usefulness,
  author = {Tiago Pimentel and Clara Meister and Ryan Cotterell},
  title = {{On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation}},
  year = {2022},
  eprint = {2205.16001},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.16001},
}

Loading related content...

Figure