Semi-supervised learning for speech recognition in the context of accent adaptation

Udhyakumar Nallasamy, Florian Metze, Tanja Schultz

TL;DR

This paper tackles the challenge of adapting Automatic Speech Recognition (ASR) systems to accented speech, which often performs poorly due to underrepresentation in training data. They introduce a novel cross-entropy based data selection method for semi-supervised learning that effectively leverages unlabeled data to improve accent adaptation, outperforming traditional self-training approaches.

What Problem Does This Paper Address?

Modern ASR systems are trained on massive datasets, but these corpora often lack diversity in accents. This leads to a significant drop in performance when encountering accented speech, which is common in real-world applications like voice search or broadcast news. While supervised adaptation (training on labeled accented data) can improve performance, collecting and transcribing such data is expensive and time-consuming. Semi-supervised learning offers a solution by utilizing readily available unlabeled data. However, existing semi-supervised methods, like self-training, often assume homogeneous training and test conditions and struggle when the unlabeled data contains multiple accents or is otherwise dissimilar to the initial training data. This paper aims to bridge this gap by developing a robust semi-supervised approach that can effectively select relevant accented data from a mixed, unlabeled corpus for accent adaptation.

Methods

The core of this paper is a novel cross-entropy based data selection method designed to identify speakers and utterances that are relevant for adapting an ASR system to a target accent.

Here's how it works:

  1. Initial Adaptation: A baseline ASR model is first adapted to the target accent using a small amount of manually transcribed data. This creates an "adapted model."
  2. Speaker Selection: The key innovation is using the difference in cross-entropy between the adapted model and the unadapted model to score speakers in the unlabeled mixed dataset. A higher score indicates that a speaker's data is more likely to belong to the target accent and will benefit from adaptation. This is a more sophisticated relevance measure than simple confidence scores.
  3. Utterance Selection: After selecting relevant speakers, utterances from these speakers are further filtered using confidence scores, similar to traditional self-training. This ensures that the selected hypotheses are also accurate.
  4. Re-training: The selected subset of unlabeled data is then used to re-train the ASR system, leading to improved performance on the target accent.

The authors compare this to self-training, which relies solely on confidence scores of decoded hypotheses from the unadapted or adapted model to select data for re-training. They also explore two scenarios: one where the mixed data has transcriptions (allowing for Viterbi alignment scores) and another where it does not.

Datasets

The experiments were conducted on two main language pairs:

  • Arabic:
    • Source Accent: Modern Standard Arabic (MSA)
    • Target Accent: Levantine Arabic
    • Training Data: ~1092 hours of Broadcast News (GALE)
    • Adaptation Data: 10 hours of Levantine Broadcast Conversations (BC)
    • Mixed Data: 20 hours of Levantine BC + 200 hours of other BC data.
  • Test Sets: ~3 hours each for MSA (Test-SRC) and Levantine (Test-TGT).
  • English:
    • Source Accent: American English (WSJ)
    • Target Accent: British English (WSJCAM0)
    • Training Data: 66 hours of American English
    • Adaptation Data: 3 hours of British English
    • Mixed Data: 12 hours of British English + 15 hours of American English (WSJO)
    • Test Sets: ~1-2.5 hours each for American (Test-SRC) and British (Test-TGT).

The "Mixed" datasets are crucial as they contain unlabeled data with varying accents and potentially other acoustic variations.

Key Insights and Ideas

  • Confidence is Not Enough for Accent Adaptation: Self-training, which relies solely on confidence scores, fails to improve performance when adapting to a new accent, especially if the unlabeled data is not homogeneous. This is because high-confidence hypotheses might still be from the source accent or irrelevant to the target accent.
  • Cross-Entropy as a Relevance Measure: The difference in cross-entropy between an adapted and an unadapted model is a powerful indicator of a speaker's relevance to the target accent. It effectively captures how much a speaker's data "changes" when viewed through the lens of the target accent model.
  • Leveraging Unlabeled Data is Key: Even with limited labeled adaptation data, semi-supervised learning using unlabeled data can significantly boost performance. The proposed cross-entropy method demonstrates this effectively.
  • Robustness to Transcription Availability: The cross-entropy method shows promise in both scenarios: with and without transcriptions in the unlabeled mixed data, highlighting its flexibility.

Key Results

The proposed cross-entropy based semi-supervised learning approach consistently outperformed self-training and the supervised adaptation baseline:

  • Arabic: Achieved improvements of 2.3% absolute (15.9% relative) over supervised adaptation when using data with transcriptions, and 0.8% absolute (2.0% relative) when using data without transcriptions.
  • English: Achieved improvements of 1.2% absolute (15.9% relative) over supervised adaptation when using data with transcriptions, and 2% absolute (13.8% relative) when using data without transcriptions.
  • Self-training: In most cases, self-training either showed no improvement or even degraded performance compared to the supervised baseline, underscoring its limitations in this context.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the "ORACLE" (perfect selection) is an upper bound and that real-world robust data selection would need to handle noise and transcription errors more effectively.
  • For Arabic, the mixed data from Broadcast Conversations (BC) might contain more Levantine speech than initially accounted for, potentially influencing results.
  • The authors note that some speakers left out of data selection might have had worse scores due to transcription errors, suggesting room for improvement in selection criteria.
  • Future Work:
    • Extend the framework to include uncertainty-based discriminative training.
  • Explore active learning techniques to intelligently select data for human annotation, further enhancing accent adaptation.

Summary updated

BibTeX
@inproceedings{nallasamy2012semi,
  author = {Udhyakumar Nallasamy and Florian Metze and Tanja Schultz},
  title = {{Semi-supervised learning for speech recognition in the context of accent adaptation}},
  booktitle = {MLSLP},
  year = {2012},
  url = {https://openreview.net/forum?id=tA08JnxXo9e},
}
Figure