Automatic Text Evaluation through the Lens of Wasserstein Barycenters

Pierre Colombo, Guillaume Staerman, Chloé Clavel, Pablo Piantanida

Abstract

Evaluating Natural Language Generation (NLG) systems requires metrics that align closely with human judgment. While embedding-based metrics like BERTScore have improved upon traditional n-gram matching, they often fail to optimally utilize the multi-layered information provided by transformer models. We present BaryScore, a metric that treats deep embeddings as probability distributions and aggregates them using Wasserstein barycenters. This approach preserves the geometric structure of the embedding space, leading to state-of-the-art performance in machine translation, summarization, and image captioning.

Introduction: Beyond String Matching

Traditional metrics such as BLEU and ROUGE rely on exact string matching, which penalizes semantically correct synonyms. The advent of contextualized embeddings (e.g., BERT, RoBERTa) led to metrics like BERTScore and MoverScore, which measure semantic similarity in vector space. However, these models typically aggregate information across multiple transformer layers using simple arithmetic means or heuristic layer selection.

The authors argue that Euclidean averaging is fundamentally flawed for high-dimensional embedding spaces. Averaging vectors often results in a point that lies outside the natural manifold of the data—akin to finding the midpoint of a curved road by drawing a straight line through the air.

Comparison of Euclidean vs. Wasserstein interpolation. Euclidean averaging (left) creates a 'ghosting' effect where peaks fade in and out, while Wasserstein interpolation (right) smoothly shifts the mass, preserving the distribution's topology.

Methodology: The Geometry of BaryScore

BaryScore addresses the limitations of Euclidean averaging by leveraging the framework of Optimal Transport. Instead of treating layer outputs as static vectors, it treats them as probability distributions.

1. Wasserstein Barycenters

The core of the metric is the calculation of a Wasserstein barycenter. For a set of distributions (representing different layers of a model), the barycenter is the distribution that minimizes the sum of Wasserstein distances to all members of the set. This "geometric mean" ensures that the aggregated representation respects the underlying manifold of the embedding space.

The BaryScore framework: (a) shows the aggregation of multiple embedding layers into a single barycentric representation by minimizing the sum of Wasserstein distances.

2. Topology Preservation and Parameter-Free Design

By operating in the Wasserstein space, BaryScore avoids the geometric distortion inherent in Euclidean methods. Furthermore, unlike MoverScore—which requires tuning a power-mean exponent—BaryScore is naturally motivated by the topology of the space and does not require manual layer selection or extensive hyperparameter tuning.

Experimental Results

The researchers evaluated BaryScore across several key NLG tasks:

  • Machine Translation: On WMT15 and WMT16 datasets, BaryScore+ (using NLI-fine-tuned models) consistently achieved the highest correlation with human judgments across nearly all language pairs.
  • Text Summarization: BaryScore demonstrated exceptional stability on the CNN/DailyMail dataset. While BERTScore often struggles to distinguish between extractive and abstractive systems, BaryScore remains a robust indicator of quality.
  • Data-to-Text and Image Captioning: BaryScore outperformed existing metrics in the majority of configurations for the WebNLG 2020 challenge and showed strong alignment with human preferences in MSCOCO image captioning.

Conclusion

BaryScore demonstrates that the geometric treatment of deep embeddings is crucial for accurate text evaluation. By replacing simple averaging with Wasserstein barycenters, the metric captures the complex linguistic features distributed across transformer layers. Future work includes extending this geometric framework to multimodal evaluation and exploring its use as a differentiable loss function for training generation models.

Summary updated

Abstract

A new metric \texttt{BaryScore} to evaluate text generation based on deep contextualized embeddings e.g., BERT, Roberta, ELMo) is introduced. This metric is motivated by a new framework relying on optimal transport tools, i.e., Wasserstein distance and barycenter. By modelling the layer output of deep contextualized embeddings as a probability distribution rather than by a vector embedding; this framework provides a natural way to aggregate the different outputs through the Wasserstein space topology. In addition, it provides theoretical grounds to our metric and offers an alternative to available solutions e.g., MoverScore and BertScore). Numerical evaluation is performed on four different tasks: machine translation, summarization, data2text generation and image captioning. Our results show that \texttt{BaryScore} outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization.
BibTeX
@inproceedings{colombo2021automatic,
  author = {Pierre Colombo and Guillaume Staerman and Chloé Clavel and Pablo Piantanida},
  title = {{Automatic Text Evaluation through the Lens of Wasserstein Barycenters}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=nHmxwUfPm8},
}
Figure