Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data

Mozhdeh Gheini, Tatiana Likhomanenko, Matthias Sperber, Hendra Setiawan

Abstract

Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing as such results in additional gains on top of the vanilla pseudo-labeling setup resulting in total improvements of up to 0.6% absolute WER and 2.2 BLEU points.

Introduction

Joint Speech Transcription and Translation (STT) is a task where a single model simultaneously generates a transcript and a translation from an audio input. This approach is particularly beneficial for user interfaces requiring concurrent display of source and target text. However, training these models is challenging due to the scarcity of "triples"—aligned audio, transcript, and translation data.

To mitigate this, researchers often employ pseudo-labeling (self-training), where a base model trained on limited labeled data generates labels for a larger unlabeled corpus. A significant hurdle in this process is domain mismatch. When unlabeled audio differs significantly from the labeled training set (e.g., in sentence length or vocabulary), the base model often produces low-quality pseudo-labels, sometimes falling into "looping" states that degrade the final model's performance.

Datasets and the Domain Mismatch Problem

The study utilizes CoVoST V2 as the supervised dataset (English audio with transcripts and translations into 15 languages) and MuST-C (TED talks) as the unlabeled set. A primary source of domain mismatch between these datasets is the distribution of audio lengths.

Density plot showing the distribution of speech duration for CoVoST (supervised) vs. MuST-C (unlabeled) datasets, highlighting the domain mismatch.

As shown in the figure above, MuST-C contains significantly longer sequences than CoVoST. When a model trained on shorter sequences encounters these longer out-of-distribution (OOD) samples, it frequently repeats words, leading to "garbage" pseudo-labels that poison the training pool.

Proposed Methods

The authors propose two primary strategies to address OOD unlabeled data without requiring additional human supervision:

1. Data-Centric Filtering

Instead of relying solely on model confidence, external metrics are used to prune poor pseudo-labels:

  • Length Ratio (KDE): The probability distribution of transcript length relative to audio length is estimated. Samples with suspiciously high or low ratios are discarded.
  • LASER Scores: A multilingual sentence encoder (LASER) evaluates the semantic similarity between the generated transcript and translation. Significant divergence indicates a likely labeling error.

2. Concatenative Data Augmentation

To prepare the model for longer OOD sequences, the authors implement a simple yet effective augmentation: randomly "stitching" together pairs of supervised samples. This "primes" the model to handle the length characteristics of the unlabeled data before the pseudo-labeling process begins.

Key Results and Insights

The proposed remedies consistently outperformed vanilla pseudo-labeling. The best configurations achieved an improvement of 0.6% absolute WER for transcription and 2.2 BLEU points for translation.

Performance comparison on CoVoST showing improvements in WER (lower is better) and BLEU (higher is better) for the best proposed setting.

Key insights include:

  • Filtering vs. Augmentation: Filtering acts as a "clean-up" mechanism, most effective in early rounds to prevent learning from catastrophic errors. Augmentation is a "pre-emptive" strategy that expands the model's capabilities from the start.
  • Efficiency: Using Ratio KDE filtering allowed the model to surpass vanilla performance in a single round, despite being trained on 10% less data due to filtering.
  • Simplicity: Concatenative augmentation proved highly effective at closing the gap toward the theoretical performance bound where perfect labels are available.

Conclusion

While this work successfully addresses length-based domain mismatch, vocabulary mismatch remains an open challenge. Future research may explore more sophisticated data selection or language model integration. These findings demonstrate that careful data-centric processing and simple augmentation can significantly enhance the robustness of self-training for joint speech tasks.

Summary updated

Abstract

Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing as such results in additional gains on top of the vanilla pseudo-labeling setup resulting in total improvements of up to 0.6% absolute WER and 2.2 BLEU points.
BibTeX
@inproceedings{gheini2023joint,
  author = {Mozhdeh Gheini and Tatiana Likhomanenko and Matthias Sperber and Hendra Setiawan},
  title = {{Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.483},
}
Figure