Abstract
Evaluating speech-to-speech translation (S2ST) systems has traditionally required intermediate text transcriptions, creating a dependency on automatic speech recognition (ASR) quality and limiting evaluation for unwritten languages. We introduce BLASER, a novel evaluation metric that operates directly on audio embeddings in a shared multimodal space. By bypassing the "textual crutch," BLASER achieves significantly higher correlation with human judgment than traditional ASR-dependent metrics like BLEU or COMET, particularly in low-resource and unwritten language settings.
Introduction
The evaluation of Speech-to-Speech Translation (S2ST) has historically been a cascaded process. To assess whether a model correctly translates source audio (e.g., Spanish) into target audio (e.g., English), researchers typically transcribe the output using an ASR system and then compare the resulting text to a reference transcript using metrics like BLEU or COMET.
This approach suffers from two primary limitations. First, evaluation accuracy is tethered to ASR performance; errors in transcription can lead to misleading quality scores. Second, many of the world’s languages—such as Hokkien or various dialects of Arabic—lack standardized writing systems, making text-based evaluation impossible. BLASER addresses these issues by evaluating speech quality directly in the embedding space.
Methodology
The foundation of BLASER is the LASER (Language-Agnostic SEntence Representations) encoder, a multilingual, multimodal model capable of mapping both text and speech into a unified embedding space. We propose two variants of the metric:
- Unsupervised (BLASERᵤ): This version computes the cosine similarity between the embeddings of the source speech, the translated output, and the reference speech.
- Supervised (BLASERₛ): This version employs a neural regressor—a 2-layer multi-layer perceptron (MLP)—trained on human quality scores. It processes a concatenated vector of features, including the raw embeddings, their element-wise products, and absolute differences, to predict a final quality score.
During training, the LASER encoders remain "frozen" to maintain the stability of the shared embedding space, ensuring the metric can be extended to new languages without retraining the core representation model.
Experimental Setup
To validate BLASER, the authors compiled a dataset of over 40,000 human annotations across seven language directions. These included standard pairs (e.g., Spanish↔English, French↔English) and a specific test for unwritten languages: Hokkien→English. Source audio was drawn from established benchmarks like MusT-C and the TAT corpus. Due to the scarcity of human-to-human speech references, high-quality Text-to-Speech (TTS) systems were used to synthesize reference audio from existing transcripts.
Results and Discussion
BLASER consistently outperformed ASR-dependent baselines in Pearson correlation with human judgments across all tested language pairs.
Key Insights
- ASR as a Bottleneck: Replacing human-written transcripts with ASR-generated ones significantly degraded the performance of traditional metrics, confirming that ASR dependency is a major weakness in current S2ST evaluation.
- Sufficiency of Speech Embeddings: Experimental results indicated that combining speech and text inputs did not improve correlation scores. Speech embeddings alone proved rich enough to capture the necessary semantic information.
- Zero-Shot Generalization: BLASERₛ demonstrated a robust ability to generalize to language directions not seen during supervised training, such as Hokkien→English, where it achieved its highest correlation scores.
Conclusion and Future Work
BLASER represents a significant shift toward text-free evaluation for speech translation. By operating directly on audio, it provides a viable path for evaluating translations in unwritten or low-resource languages.
However, several challenges remain. Current implementations operate at the sentence level; future work will focus on developing corpus-level metrics. Additionally, while global embeddings capture broad semantics, they can be insensitive to granular errors, such as incorrect numerical values. Future iterations will also explore the use of natural human-spoken references and better accounting for the prosodic naturalness of synthesized speech.
Summary updated