BLASER 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation

David Dale, Marta R. Costa-jussàEMNLP 2024 · DOI

TL;DR

BLASER 2.0 is a massively multilingual, modality-agnostic metric designed to evaluate translation quality across both speech and text. By leveraging the SONAR embedding space, it provides highly interpretable scores that excel at reference-free quality estimation, hallucination detection, and filtering noisy training data across hundreds of languages.

Introduction: The Challenges of Multimodal Translation Evaluation

Evaluating machine translation (MT) is notoriously difficult, particularly as we move beyond simple text-to-text paradigms. Traditional metrics like BLEU or ChrF rely on "surface-form" overlap—counting how many words or characters a translation shares with a human-provided reference. This approach faces three major hurdles:

  1. Modality Mismatch: Computing n-gram overlap for speech-to-speech translation (S2ST) typically requires an intermediate Automatic Speech Recognition (ASR) step, which introduces errors and complicates the pipeline.
  2. Semantic Flexibility: A translation can be perfectly accurate in meaning but use entirely different vocabulary than the reference, resulting in unfairly low scores from overlap-based metrics.
  3. The Reference Gap: In many real-world scenarios, such as live translation or web-scale data mining, human-provided "gold standard" references are unavailable. This necessitates Quality Estimation (QE)—the ability to score a translation using only the source input.

BLASER 2.0 addresses these challenges by shifting evaluation into a shared, multilingual, and multimodal embedding space where "meaning" is represented numerically, independent of the input modality (audio or text).

Methods: Leveraging the SONAR Embedding Space

The foundational component of BLASER 2.0 is the SONAR embedding space. SONAR is designed to be modality-agnostic, covering 202 text languages and 57 speech languages. In this space, an English text sentence and its corresponding French speech translation should ideally map to very similar vectors.

Architecture of BLASER 2.0 utilizing SONAR embeddings for text and speech.

The authors propose two primary versions of the metric:

  • BLASER 2.0-Ref (Reference-based): This version utilizes the embeddings of the source ($e_s$), the translation ($e_t$), and the reference ($e_r$). These vectors are concatenated along with their element-wise products and absolute differences to capture nuanced relationships.
  • BLASER 2.0-QE (Reference-free): This version relies solely on the source ($e_s$) and translation ($e_t$) embeddings, making it suitable for scenarios where references are missing.

These concatenated vectors are fed into a simple 3-layer Multilayer Perceptron (MLP). The model is trained to predict XSTS (Cross-lingual Semantic Textual Similarity) scores—a human-annotated scale ranging from 1 (completely different meaning) to 5 (perfectly preserved meaning and style).

Key Insights

The Power of Quality Estimation (QE)

A surprising finding was that the reference-free (QE) version often performs as well as, or even better than, the reference-based version when the available references are noisy or misaligned. This suggests that the source-to-translation relationship within the SONAR space is highly reliable for assessing semantic equivalence.

Hallucination Detection

BLASER 2.0-QE proved remarkably effective at identifying "hallucinations"—instances where a model generates fluent-sounding but completely unrelated text. It achieved an ROC AUC of approximately 0.98 on WMT23 hallucination detection tasks, rivaling much larger, specialized models like CometKiwi.

Interpretability and Universal Thresholds

Unlike "perplexity"-based filtering, which is sensitive to specific tokenizers and languages, BLASER 2.0 scores are on a fixed 1-5 scale. This provides a universal "quality threshold" that works consistently across different languages and modalities, simplifying the process of cleaning large-scale datasets.

Key Results

BLASER 2.0-Ref achieved a 0.45 average Pearson correlation with human judgments across all test partitions. This represents a significant improvement over the original BLASER (v1.0), which scored 0.26 on text-to-text tasks.

Correlation between human-annotated XSTS scores and BLASER 2.0-Ref scores.

Furthermore, the metric's utility in data filtering was demonstrated using the NLLB dataset. By retaining only the top 50% of sentence pairs as ranked by BLASER 2.0-QE, the resulting MT model's performance improved by approximately 1 BLEU point. This result is comparable to state-of-the-art internal filtering methods used by NLLB but offers greater extensibility to new languages and modalities.

Limitations and Future Work

While BLASER 2.0 is a powerful tool, it has specific limitations:

  • Accuracy vs. Fluency: The XSTS protocol prioritizes meaning preservation. Consequently, the metric may not heavily penalize translations that sound robotic or contain minor audio glitches if the core message remains intact.
  • SONAR Dependency: As a "downstream" application of SONAR, BLASER 2.0 inherits any biases or "blind spots" present in the underlying encoders, particularly for low-resource languages.
  • Audio Constraints: Current speech encoders are optimized for segments shorter than 30 seconds, requiring prior segmentation for long-form content like lectures or podcasts.

Summary updated

BibTeX
@inproceedings{dale2024blaser,
  author = {David Dale and Marta R. Costa-jussà},
  title = {{BLASER 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.943},
}
Figure