Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data

Mozhdeh Gheini, Tatiana Likhomanenko, Matthias Sperber, Hendra Setiawan

Abstract

Joint speech transcription and translation (STT) models often suffer from a lack of high-quality "triple" data (audio, transcript, and translation). While pseudo-labeling (PL) offers a path to leverage unlabeled data, performance degrades when the unlabeled data is out-of-distribution (OOD) relative to the training set. This paper introduces data-centric filtering and augmentation techniques to address domain mismatch, specifically targeting sequence length and semantic consistency. Our methods yield significant improvements in both Word Error Rate (WER) and BLEU scores across English-to-German and English-to-Chinese benchmarks.

Introduction: The Challenge of OOD Pseudo-Labeling

Joint STT models are multitasking systems that produce both a transcript and a translation from a single audio input. However, the scarcity of labeled triples ($audio, transcript, translation$) necessitates semi-supervised approaches like pseudo-labeling. In a typical PL pipeline, a base model labels a large unlabeled dataset, which is then used for retraining.

A critical bottleneck identified in this research is the "domain mismatch" between supervised training data (e.g., CoVoST) and unlabeled OOD data (e.g., MuST-C). As shown in Figure 1, the distribution of audio lengths can vary significantly between datasets. When a model trained on short segments encounters longer OOD audio, it often fails, producing low-quality pseudo-labels characterized by repetitive "looping" text.

Density distribution of speech durations in CoVoST (supervised) and MuST-C (unlabeled) datasets.

Methods: Filtering and Augmentation

To mitigate the impact of OOD data, the authors propose two primary strategies:

1. Data-Centric Filtering

Rather than accepting all generated pseudo-labels, the authors filter them based on two criteria:

  • Ratio KDE: Using Kernel Density Estimation, the model learns the expected relationship between audio duration and transcript length from the supervised data. Pseudo-labels with anomalous length ratios are discarded.
  • LASER Score: A pre-trained LASER multilingual sentence encoder evaluates the semantic similarity between the generated transcript and translation. Low similarity scores indicate likely transcription or translation errors.

2. Data Augmentation via Concatenation

To bridge the gap in sequence lengths, the authors employ a simple augmentation technique: randomly concatenating pairs of supervised samples. This creates synthetic long-form data that better prepares the model for the longer sequences found in OOD datasets like MuST-C.

Key Results

The proposed filtering and augmentation techniques consistently outperform standard pseudo-labeling baselines. By addressing the distribution shift, the model generates more reliable labels and achieves higher performance ceilings.

Performance comparison of different training settings on CoVoST WER (lower is better) and BLEU (higher is better).

As illustrated in Figure 2, the "Best Setting" (incorporating filtering and augmentation) achieves superior results. Specifically:

  • En–De: Improvements of 0.4% absolute WER and 2.1 BLEU points over vanilla PL.
  • En–Zh: Improvements of 0.6% absolute WER and 2.2 BLEU points.

The "Ratio KDE" method proved particularly efficient, allowing the model to reach higher performance levels with fewer samples by prioritizing high-quality labels early in the training process.

Conclusion

This work demonstrates that the effectiveness of pseudo-labeling for joint STT is heavily dependent on managing domain mismatch. By using simple, supervision-free signals like audio-to-text length ratios and semantic consistency checks, researchers can significantly improve the quality of pseudo-labels. Future work may explore handling vocabulary mismatches and incorporating incomplete data pairs (ASR or ST only) to further enhance model robustness.

Summary updated

Abstract

Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing as such results in additional gains on top of the vanilla pseudo-labeling setup resulting in total improvements of up to 0.6% absolute WER and 2.2 BLEU points.
BibTeX
@inproceedings{gheini2023joint,
  author = {Mozhdeh Gheini and Tatiana Likhomanenko and Matthias Sperber and Hendra Setiawan},
  title = {{Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.483},
}
Figure