Abstract
A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality. Yet, there is a dearth of such metrics, which inhibits the rapid and efficient progress of language generators. One exception is the recently proposed Mauve. In theory, Mauve measures an information-theoretic divergence between two probability distributions over strings: one representing the language generator under evaluation; the other representing the true natural language distribution. Mauve's authors argue that its success comes from the qualitative properties of their proposed divergence. Yet in practice, as this divergence is uncomputable, Mauve approximates it by measuring the divergence between multinomial distributions over clusters instead, where cluster assignments are attained by grouping strings based on a pre-trained language model's embeddings. As we show, however, this is not a tight approximation -- in either theory or practice. This begs the question: why does Mauve work so well? In this work, we show that Mauve was right for the wrong reasons, and that its newly proposed divergence is not necessary for its high performance. In fact, classical divergences paired with its proposed cluster-based approximation may actually serve as better evaluation metrics. We finish the paper with a probing analysis; this analysis leads us to conclude that -- by encoding syntactic- and coherence-level features of text, while ignoring surface-level features -- such cluster-based substitutes to string distributions may simply be better for evaluating state-of-the-art language generators.
Introduction
Evaluating the output of modern text generators remains a significant hurdle in NLP. Traditional metrics like perplexity or BLEU often fail to align with human perceptions of quality, interestingness, or fluency. Recently, MAUVE has emerged as a gold-standard metric, theoretically measuring the divergence between the distribution of human language and a model's generated text. However, because the true distribution of all possible strings is uncomputable, MAUVE utilizes a practical workaround: mapping text into a high-dimensional embedding space, clustering those embeddings, and calculating divergence between the resulting cluster distributions.
This research investigates whether MAUVE's success stems from its specific mathematical "Area Under the Curve" (AUC) divergence or if it is simply a byproduct of operating on semantic clusters rather than raw strings.
Right for the Wrong Reasons
The core theoretical justification for MAUVE is that it approximates string-level divergence. However, our analysis reveals that this approximation is not tight. In fact, as shown in Figure 1, the cluster-based probabilities ($\hat{q}_c$) actually correlate negatively with the estimated string-level probabilities ($\hat{q}_w$) across various model sizes. This suggests that MAUVE succeeds not because it accurately models string distributions, but for other qualitative reasons related to the clustering process itself.
The Power of Discretization
To isolate the impact of clustering, we conducted an ablation study comparing traditional string-based metrics (using n-gram models) against cluster-based versions using several classical divergence measures: Forward KL, Backward KL, Jensen-Shannon (JS), and Exponentiated KL. All metrics utilized GPT-2 XL embeddings followed by PCA and K-means for clustering.
The results were striking: every divergence measure tested saw a massive boost in correlation with human judgment when switched to a cluster-based implementation. As seen in Figure 2, cluster-based metrics ($\Delta(p_c, q_c)$) consistently outperform their string-based counterparts ($\Delta(p_w, q_w)$) across three key human judgment categories: interestingness, sensibility, and human-likeness.
Notably, simpler metrics like cluster-based Jensen-Shannon (JS) and Backward KL achieved slightly higher correlations with human sensibility and human-likeness than the original MAUVE (AUC) metric. These cluster-based metrics also exhibited much lower variance across different random seeds, providing a more stable evaluation tool for researchers.
Why Clusters Work: Denoising and Coherence
Our probing analysis suggests that clusters act as a "denoising filter." By forcing high-dimensional embeddings into discrete clusters, we effectively group similar meanings while ignoring surface-level features. Perturbation testing showed that cluster-based metrics are robust to minor grammatical noise (like removing articles) but highly sensitive to structural issues that destroy coherence (like shuffling sentence order). This alignment with high-level semantic and syntactic features—rather than word-level overlap—is likely why these metrics correlate so well with human intuition.
Conclusion
MAUVE's high performance is primarily due to its use of embedding-based clusters rather than its specific divergence math. This finding opens the door for simpler, more robust evaluation metrics based on classical information theory applied to semantic clusters. Future work should explore discretization algorithms beyond K-means to better capture specific linguistic attributes like factual consistency and logical flow.
Summary updated