Abstract
End-to-end speech translation (E2E-ST) faces significant challenges due to data scarcity and the inherent "modal gap" between continuous speech signals and discrete text tokens. We present FCGCL (Fine- and Coarse-Granularity Contrastive Learning), a framework designed to explicitly align speech and text representations. By leveraging a frozen, pretrained machine translation (MT) model as a teacher and introducing a whitening operation to combat representation anisotropy, FCGCL achieves state-of-the-art performance on the MuST-C benchmark with an average BLEU of 29.0.
Introduction
Traditional cascaded speech translation systems suffer from latency and error propagation. While end-to-end models offer a more streamlined alternative, they are hindered by the lack of large-scale paired speech-translation datasets and the difficulty of mapping high-dimensional audio to symbolic text. Furthermore, text embeddings in neural models often exhibit anisotropy, where vectors are clustered in a narrow cone, reducing the effectiveness of standard similarity metrics. FCGCL addresses these issues through explicit knowledge transfer and multi-level alignment.
Methodology: The FCGCL Framework
The FCGCL framework utilizes a pretrained MT model to guide the training of an ST model without updating the MT parameters. This stability allows the ST model to learn robust translation logic from a fixed, high-quality target.
Multi-Granularity Contrastive Learning
The core of FCGCL is the contrastive module, which operates at two levels:
- Coarse-Granularity: Aligns sentence-level representations by comparing the average vectors of speech and text sequences.
- Fine-Granularity: Employs an unsupervised maximum similarity method to align individual speech frames with corresponding text tokens, providing the detailed mapping necessary for high-quality generation.
The Whitening Operation
To resolve the anisotropy problem in text embeddings, FCGCL applies a linear whitening transformation. This process maps the "cone-shaped" distribution of embeddings into an isotropic standard normal distribution, significantly improving the discriminative power of the contrastive loss by making cosine similarity a more reliable measure.
Experimental Results
The model was evaluated on the MuST-C dataset, covering English audio translated into eight languages. FCGCL achieved an average BLEU score of 29.0, outperforming previous E2E-ST models and even surpassing strong cascaded baselines in several language pairs (e.g., English-to-German).
Ablation studies confirmed that both fine-grained alignment and the whitening operation are critical for performance. As shown in the figure above, the whitening operation consistently provides a performance boost. However, the model exhibits sensitivity to the size of the negative sample queue, with performance peaking at approximately 1,000 samples before declining, suggesting a capacity limit for the whitening transformation.
Conclusion
FCGCL demonstrates that explicit, multi-level alignment with a frozen MT teacher can effectively bridge the modal gap in speech translation. The inclusion of a whitening operation is essential for maximizing the utility of contrastive learning in the presence of anisotropic text representations. Future work will explore intra-modal contrastive losses within the speech domain to further improve representation uniformity.
Summary updated