Dual supervised learning for non-native speech recognition

Kacper Radzikowski, Robert Nowak, Le Wang, Osamu YoshieEURASIP 2019 · DOI

Abstract

Current automatic speech recognition (ASR) systems achieve over 90–95% accuracy, depending on the methodology applied and datasets used. However, the level of accuracy decreases significantly when the same ASR system is used by a non-native speaker of the language to be recognized. At the same time, the volume of labeled datasets of non-native speech samples is extremely limited both in size and in the number of existing languages. This problem makes it difficult to train or build sufficiently accurate ASR systems targeted at non-native speakers, which, consequently, calls for a different approach that would make use of vast amounts of large unlabeled datasets. In this paper, we address this issue by employing dual supervised learning (DSL) and reinforcement learning with policy gradient methodology. We tested DSL in a warm-start approach, with two models trained beforehand, and in a semi warm-start approach with only one of the two models pre-trained. The experiments were conducted on English language pronounced by Japanese and Polish speakers. The results of our experiments show that creating ASR systems with DSL can achieve an accuracy comparable to traditional methods, while simultaneously making use of unlabeled data, which obviously is much cheaper to obtain and comes in larger sizes.

Introduction: The Non-Native Accuracy Gap

While modern Automatic Speech Recognition (ASR) systems perform exceptionally well for native speakers, their performance often plummets when encountering non-native accents. This "accuracy gap" is primarily due to the phonetic interference from a speaker's mother tongue, creating acoustic profiles that differ significantly from the native training data. Bridging this gap typically requires massive labeled datasets of non-native speech, which are both rare and expensive to produce for every possible accent-language pair.

Methodology: Leveraging Duality

To overcome the scarcity of labeled data, this research utilizes Dual Supervised Learning (DSL). The core premise is that speech-to-text (STT) and text-to-speech (TTS) are reciprocal tasks. By coupling them, the system can learn from unlabeled data through self-correction.

The architecture integrates four primary components:

  1. Language Model ($M_L$): Pre-trained on text to judge the naturalness of sentences.
  2. Acoustic Model ($M_S$): Pre-trained on unlabeled audio to estimate the probability of sound sequences.
  3. STT Model ($M_{STT}$): The target model for recognition.
  4. TTS Model ($M_{TTS}$): A synthesis model that provides feedback to the STT model.

The Feedback Loop Mechanism

The system operates through two closed loops. Loop L (Text-centric) begins with a text sentence, generates audio via TTS, and then reconstructs the text via STT. Conversely, Loop S (Audio-centric) starts with raw audio, transcribes it via STT, and reconstructs the audio via TTS.

Feedback loop design illustrating the text-centric Loop L, where text is synthesized into speech and then recognized back into text.

In these loops, the pre-trained $M_L$ and $M_S$ act as "judges," providing reward signals via policy gradients to update the trainable $M_{STT}$ and $M_{TTS}$ models. This allows the system to improve by ensuring that its internal "translations" between audio and text remain consistent with known language and acoustic patterns.

Training Process and Workflow

The training follows an iterative process where rewards are calculated based on the consistency of the reconstructions in both loops. A "warm-start" strategy is employed, where models are initially pre-trained on a small amount of labeled data to prevent the reinforcement learning process from converging on incorrect associations.

Flowchart of the DSL process, showing the initialization, iterative loop execution, reward calculation, and model updates.

Experimental Results

The DSL framework was tested on English speech from Japanese and Polish speakers across three architectural setups (RNN, LSTM, and 3-gram based). The LSTM-based configuration yielded the most robust performance. In a warm-start configuration, the system achieved:

  • 89.43% accuracy for Japanese-accented English.
  • 88.14% accuracy for Polish-accented English.

Notably, the Japanese results outperformed a traditional supervised baseline (87.24%) that required significantly more labeled data. This demonstrates that for niche accents, leveraging unlabeled data through dual learning is a highly effective strategy. However, this efficiency in data comes at a computational cost; the DSL model required approximately five weeks of training on a GTX 1080 Ti, compared to four days for the supervised baseline.

Conclusion

Dual Supervised Learning offers a powerful solution to the data scarcity problem in non-native ASR. While the computational cost is higher, the ability to achieve high accuracy using cheap, unlabeled data makes it a viable path for developing robust recognition systems for diverse global accents. Future work will focus on refining reward functions to move toward fully unsupervised models.

Summary updated

Abstract

Current automatic speech recognition (ASR) systems achieve over 90–95% accuracy, depending on the methodology applied and datasets used. However, the level of accuracy decreases significantly when the same ASR system is used by a non-native speaker of the language to be recognized. At the same time, the volume of labeled datasets of non-native speech samples is extremely limited both in size and in the number of existing languages. This problem makes it difficult to train or build sufficiently accurate ASR systems targeted at non-native speakers, which, consequently, calls for a different approach that would make use of vast amounts of large unlabeled datasets. In this paper, we address this issue by employing dual supervised learning (DSL) and reinforcement learning with policy gradient methodology. We tested DSL in a warm-start approach, with two models trained beforehand, and in a semi warm-start approach with only one of the two models pre-trained. The experiments were conducted on English language pronounced by Japanese and Polish speakers. The results of our experiments show that creating ASR systems with DSL can achieve an accuracy comparable to traditional methods, while simultaneously making use of unlabeled data, which obviously is much cheaper to obtain and comes in larger sizes.
BibTeX
@inproceedings{radzikowski2019dual,
  author = {Kacper Radzikowski and Robert Nowak and Le Wang and Osamu Yoshie},
  title = {{Dual supervised learning for non-native speech recognition}},
  booktitle = {EURASIP},
  year = {2019},
  doi = {10.1186/s13636-018-0146-4},
}
Figure