Abstract
Joint speech transcription and translation (STT) models often suffer from a lack of high-quality "triple" data (audio, transcript, and translation). While pseudo-labeling (PL) offers a path to leverage unlabeled data, performance degrades when the unlabeled data is out-of-distribution (OOD) relative to the training set. This paper introduces data-centric filtering and augmentation techniques to address domain mismatch, specifically targeting sequence length and semantic consistency. Our methods yield significant improvements in both Word Error Rate (WER) and BLEU scores across English-to-German and English-to-Chinese benchmarks.
Introduction: The Challenge of OOD Pseudo-Labeling
Joint STT models are multitasking systems that produce both a transcript and a translation from a single audio input. However, the scarcity of labeled triples ($audio, transcript, translation$) necessitates semi-supervised approaches like pseudo-labeling. In a typical PL pipeline, a base model labels a large unlabeled dataset, which is then used for retraining.
A critical bottleneck identified in this research is the "domain mismatch" between supervised training data (e.g., CoVoST) and unlabeled OOD data (e.g., MuST-C). As shown in Figure 1, the distribution of audio lengths can vary significantly between datasets. When a model trained on short segments encounters longer OOD audio, it often fails, producing low-quality pseudo-labels characterized by repetitive "looping" text.
Methods: Filtering and Augmentation
To mitigate the impact of OOD data, the authors propose two primary strategies:
1. Data-Centric Filtering
Rather than accepting all generated pseudo-labels, the authors filter them based on two criteria:
- Ratio KDE: Using Kernel Density Estimation, the model learns the expected relationship between audio duration and transcript length from the supervised data. Pseudo-labels with anomalous length ratios are discarded.
- LASER Score: A pre-trained LASER multilingual sentence encoder evaluates the semantic similarity between the generated transcript and translation. Low similarity scores indicate likely transcription or translation errors.
2. Data Augmentation via Concatenation
To bridge the gap in sequence lengths, the authors employ a simple augmentation technique: randomly concatenating pairs of supervised samples. This creates synthetic long-form data that better prepares the model for the longer sequences found in OOD datasets like MuST-C.
Key Results
The proposed filtering and augmentation techniques consistently outperform standard pseudo-labeling baselines. By addressing the distribution shift, the model generates more reliable labels and achieves higher performance ceilings.
As illustrated in Figure 2, the "Best Setting" (incorporating filtering and augmentation) achieves superior results. Specifically:
- En–De: Improvements of 0.4% absolute WER and 2.1 BLEU points over vanilla PL.
- En–Zh: Improvements of 0.6% absolute WER and 2.2 BLEU points.
The "Ratio KDE" method proved particularly efficient, allowing the model to reach higher performance levels with fewer samples by prioritizing high-quality labels early in the training process.
Conclusion
This work demonstrates that the effectiveness of pseudo-labeling for joint STT is heavily dependent on managing domain mismatch. By using simple, supervision-free signals like audio-to-text length ratios and semantic consistency checks, researchers can significantly improve the quality of pseudo-labels. Future work may explore handling vocabulary mismatches and incorporating incomplete data pairs (ASR or ST only) to further enhance model robustness.
Summary updated