Abstract
A new metric, BaryScore, is introduced to evaluate text generation based on deep contextualized embeddings (e.g., BERT, RoBERTa, ELMo). This metric is motivated by a framework relying on optimal transport tools: Wasserstein distance and barycenters. By modeling the layer outputs of deep contextualized embeddings as probability distributions rather than simple vector embeddings, this framework provides a natural way to aggregate different outputs through the Wasserstein space topology. This approach offers a theoretically grounded alternative to existing solutions like MoverScore and BERTScore. Numerical evaluation across four tasks—machine translation, summarization, data-to-text generation, and image captioning—demonstrates that BaryScore outperforms other BERT-based metrics and exhibits more consistent behavior, particularly in text summarization.
Introduction: The Challenge of NLG Evaluation
Evaluating Natural Language Generation (NLG)—such as machine translation or AI-generated summaries—is a persistent challenge in NLP. Traditional metrics like BLEU rely on exact n-gram overlaps, which fail to capture semantic meaning, synonyms, or paraphrasing.
Recent "untrained" metrics, such as BERTScore and MoverScore, addressed this by utilizing contextual embeddings from transformer models. However, these models consist of multiple layers, each capturing distinct linguistic features (e.g., syntax vs. semantics). Current metrics handle these layers sub-optimally: they either select a single "best" layer (BERTScore), which is often arbitrary, or they average layers in Euclidean space (MoverScore). The authors argue that Euclidean averaging creates a "geometrical discrepancy" because the final distance calculation typically uses Wasserstein distance (Earth Mover’s Distance), which operates on a fundamentally different geometry.
Methodology: Leveraging Optimal Transport
BaryScore introduces a framework where Optimal Transport (OT) is used not just for the final comparison between candidate and reference, but for the internal aggregation of model layers.
1. Representing Layers as Distributions
Instead of treating a layer's output as a static vector, BaryScore treats the embeddings of a sentence across different layers as a set of probability distributions. This allows the metric to account for the uncertainty and varying focus of different transformer layers.
2. Wasserstein Barycenters for Aggregation
To "average" these layers, the authors calculate a Wasserstein barycenter. Unlike a Euclidean mean, which simply averages coordinates, a Wasserstein barycenter finds a central distribution that minimizes the "work" required to transport mass from all individual layers to a central point.
As shown in the figure above, Euclidean interpolation fails to preserve the structure of the distribution during the transition, whereas Wasserstein interpolation respects the underlying geometry. By using the Wasserstein barycenter, BaryScore avoids the geometric distortion inherent in Euclidean averaging.
3. The BaryScore Metric
To compare a candidate sentence to a reference, the metric computes the Wasserstein barycenter for both, then calculates the Wasserstein distance between these two central distributions. The authors also introduce BaryScore+, which utilizes a BERT model fine-tuned on Natural Language Inference (NLI) data to further sharpen semantic sensitivity.
Key Insights and Results
The researchers evaluated BaryScore across four distinct NLG tasks: Machine Translation (WMT15/16), Summarization (CNN/DailyMail), Data2Text (WebNLG 2020), and Image Captioning (MSCOCO).
- Geometry Matters: The core innovation is the alignment of the aggregation geometry with the comparison geometry. Euclidean averaging "distorts" the information before the final OT calculation.
- Parameter-Free Robustness: Unlike MoverScore, which requires tuning a specific power mean hyperparameter, BaryScore provides a theoretically grounded, parameter-free way to leverage all layers.
- Superior Correlation: BaryScore+ consistently achieved the highest correlation with human judgments in Machine Translation across almost all language pairs.
- Stability in Summarization: A striking finding was BaryScore's consistency in text summarization. While previous BERT-based metrics often showed low correlation when evaluating extractive systems, BaryScore remained robust.
- Competitive Performance: Despite the complexity of barycenter calculations, the implementation (using the POT library) achieves speeds comparable to MoverScore, processing approximately 180 sentence pairs per second on an NVIDIA V100.
Conclusion and Future Work
BaryScore demonstrates that the geometric lens through which we view model layers significantly impacts evaluation quality. While the metric is dependent on the quality of the underlying pre-trained model and carries more computational overhead than n-gram metrics, its superior alignment with human judgment makes it a powerful tool for NLG research. Future work may explore multimodal expansions, such as combining text and image embeddings directly within the Wasserstein framework, or utilizing BaryScore as a loss function for training models.
Summary updated