Abstract
Evaluating Natural Language Generation (NLG) systems requires metrics that align closely with human judgment. While embedding-based metrics like BERTScore have improved upon traditional n-gram matching, they often fail to optimally utilize the multi-layered information provided by transformer models. We present BaryScore, a metric that treats deep embeddings as probability distributions and aggregates them using Wasserstein barycenters. This approach preserves the geometric structure of the embedding space, leading to state-of-the-art performance in machine translation, summarization, and image captioning.
Introduction: Beyond String Matching
Traditional metrics such as BLEU and ROUGE rely on exact string matching, which penalizes semantically correct synonyms. The advent of contextualized embeddings (e.g., BERT, RoBERTa) led to metrics like BERTScore and MoverScore, which measure semantic similarity in vector space. However, these models typically aggregate information across multiple transformer layers using simple arithmetic means or heuristic layer selection.
The authors argue that Euclidean averaging is fundamentally flawed for high-dimensional embedding spaces. Averaging vectors often results in a point that lies outside the natural manifold of the data—akin to finding the midpoint of a curved road by drawing a straight line through the air.
Methodology: The Geometry of BaryScore
BaryScore addresses the limitations of Euclidean averaging by leveraging the framework of Optimal Transport. Instead of treating layer outputs as static vectors, it treats them as probability distributions.
1. Wasserstein Barycenters
The core of the metric is the calculation of a Wasserstein barycenter. For a set of distributions (representing different layers of a model), the barycenter is the distribution that minimizes the sum of Wasserstein distances to all members of the set. This "geometric mean" ensures that the aggregated representation respects the underlying manifold of the embedding space.
2. Topology Preservation and Parameter-Free Design
By operating in the Wasserstein space, BaryScore avoids the geometric distortion inherent in Euclidean methods. Furthermore, unlike MoverScore—which requires tuning a power-mean exponent—BaryScore is naturally motivated by the topology of the space and does not require manual layer selection or extensive hyperparameter tuning.
Experimental Results
The researchers evaluated BaryScore across several key NLG tasks:
- Machine Translation: On WMT15 and WMT16 datasets, BaryScore+ (using NLI-fine-tuned models) consistently achieved the highest correlation with human judgments across nearly all language pairs.
- Text Summarization: BaryScore demonstrated exceptional stability on the CNN/DailyMail dataset. While BERTScore often struggles to distinguish between extractive and abstractive systems, BaryScore remains a robust indicator of quality.
- Data-to-Text and Image Captioning: BaryScore outperformed existing metrics in the majority of configurations for the WebNLG 2020 challenge and showed strong alignment with human preferences in MSCOCO image captioning.
Conclusion
BaryScore demonstrates that the geometric treatment of deep embeddings is crucial for accurate text evaluation. By replacing simple averaging with Wasserstein barycenters, the metric captures the complex linguistic features distributed across transformer layers. Future work includes extending this geometric framework to multimodal evaluation and exploring its use as a differentiable loss function for training generation models.
Summary updated