FCGCL: Fine- and Coarse-Granularity Contrastive Learning for Speech Translation

Hao Zhang, Nianwen Si, Yaqi Chen, Zhen Li, Tong Niu, Xukui Yang, Dan QuEMNLP 2022 · DOI

Abstract

End-to-end speech translation (E2E-ST) faces significant challenges due to data scarcity and the inherent "modal gap" between continuous speech signals and discrete text tokens. We present FCGCL (Fine- and Coarse-Granularity Contrastive Learning), a framework designed to explicitly align speech and text representations. By leveraging a frozen, pretrained machine translation (MT) model as a teacher and introducing a whitening operation to combat representation anisotropy, FCGCL achieves state-of-the-art performance on the MuST-C benchmark with an average BLEU of 29.0.

Introduction

Traditional cascaded speech translation systems suffer from latency and error propagation. While end-to-end models offer a more streamlined alternative, they are hindered by the lack of large-scale paired speech-translation datasets and the difficulty of mapping high-dimensional audio to symbolic text. Furthermore, text embeddings in neural models often exhibit anisotropy, where vectors are clustered in a narrow cone, reducing the effectiveness of standard similarity metrics. FCGCL addresses these issues through explicit knowledge transfer and multi-level alignment.

Methodology: The FCGCL Framework

The FCGCL framework utilizes a pretrained MT model to guide the training of an ST model without updating the MT parameters. This stability allows the ST model to learn robust translation logic from a fixed, high-quality target.

The FCGCL architecture, featuring the contrastive module for fine- and coarse-grained alignment between the MT and ST encoders.

Multi-Granularity Contrastive Learning

The core of FCGCL is the contrastive module, which operates at two levels:

  1. Coarse-Granularity: Aligns sentence-level representations by comparing the average vectors of speech and text sequences.
  2. Fine-Granularity: Employs an unsupervised maximum similarity method to align individual speech frames with corresponding text tokens, providing the detailed mapping necessary for high-quality generation.

The Whitening Operation

To resolve the anisotropy problem in text embeddings, FCGCL applies a linear whitening transformation. This process maps the "cone-shaped" distribution of embeddings into an isotropic standard normal distribution, significantly improving the discriminative power of the contrastive loss by making cosine similarity a more reliable measure.

Experimental Results

The model was evaluated on the MuST-C dataset, covering English audio translated into eight languages. FCGCL achieved an average BLEU score of 29.0, outperforming previous E2E-ST models and even surpassing strong cascaded baselines in several language pairs (e.g., English-to-German).

Impact of the whitening operation on BLEU scores relative to the number of negative samples.

Ablation studies confirmed that both fine-grained alignment and the whitening operation are critical for performance. As shown in the figure above, the whitening operation consistently provides a performance boost. However, the model exhibits sensitivity to the size of the negative sample queue, with performance peaking at approximately 1,000 samples before declining, suggesting a capacity limit for the whitening transformation.

Conclusion

FCGCL demonstrates that explicit, multi-level alignment with a frozen MT teacher can effectively bridge the modal gap in speech translation. The inclusion of a whitening operation is essential for maximizing the utility of contrastive learning in the presence of anisotropic text representations. Future work will explore intra-modal contrastive losses within the speech domain to further improve representation uniformity.

Summary updated

BibTeX
@inproceedings{zhang2022fcgcl,
  author = {Hao Zhang and Nianwen Si and Yaqi Chen and Zhen Li and Tong Niu and Xukui Yang and Dan Qu},
  title = {{FCGCL: Fine- and Coarse-Granularity Contrastive Learning for Speech Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.222},
}
Figure