Self-Imitating Feedback Generation Using GAN for Computer-Assisted Pronunciation Training

Seung Hee Yang, Minhwa ChungInterspeech 2019

TL;DR

This paper introduces a novel approach to computer-assisted pronunciation training (CAPT) by using Generative Adversarial Networks (GANs) to create "self-imitating" feedback. Instead of just correcting pronunciation, this method transforms a learner's speech into a version that sounds like their own speech but with native-like pronunciation, aiming to improve learning through imitation.

What Problem Does This Paper Address?

Learning a new language involves mastering not only vocabulary and grammar but also pronunciation. Computer-Assisted Pronunciation Training (CAPT) systems aim to help learners improve their accent. A key challenge in CAPT is generating effective corrective feedback. Traditional methods often rely on teachers to record corrected versions, which is time-consuming, or use automated techniques like prosodic transplantation (e.g., PSOLA). While these methods can improve suprasegmental features (like intonation and rhythm), they often fall short in correcting segmental errors (individual sounds) and achieving a natural, imitable sound. The authors observed that spectrograms (visual representations of speech) of native and non-native speakers show distinct differences, suggesting that visual-based AI techniques like GANs could be leveraged to bridge this gap and generate more nuanced, self-imitative feedback.

Methods

The core of this work is the application of Generative Adversarial Networks (GANs) for speech spectrogram transformation. The authors propose a GAN-based framework that takes a non-native speaker's speech spectrogram as input and generates a "corrected" spectrogram that mimics native speech characteristics.

Key methodological aspects include:

  • Spectrogram Representation: Speech is converted into spectrograms, which are treated as images. This allows the use of image-generation GAN architectures.
  • Conditional GAN (CGAN): To ensure the generated spectrogram corresponds to the input utterance, a conditional GAN (specifically, a Pix2Pix-like architecture) is used. This means the generator is conditioned on the input spectrogram.
  • Cycle-Consistent Adversarial Networks (CycleGAN): To address the challenge of having many possible "golden references" (native speaker utterances) for a single non-native utterance, and to ensure the output preserves the global structure of the original speech, the authors incorporate cycle consistency loss. This loss encourages the network to be able to translate from non-native to native and back to the original non-native, preventing arbitrary transformations.
  • Two-Stage Training: The model is trained in two phases: first, using a Pix2Pix-like framework for paired native and non-native spectrograms (for conditional generation), and then using a CycleGAN framework for unpaired spectrograms (to enforce cycle consistency).
  • Audio Conversion: After spectrogram generation, the Griffin-Lim algorithm is used to convert the generated spectrograms back into audio waveforms for playback as feedback.

Datasets

  • Training Data: The L2KSC (L2 as Korean Speech Corpus) is used. This corpus contains 217 non-native speakers (with 27 mother tongues) and 107 native speakers. Each speaker read 300 short utterances. For training, approximately 97,200 spectrogram images of native and non-native utterances were used.
  • Testing Data: A held-out set of 162 spectrograms was used for testing.

Key Insights and Ideas

  • Spectrograms as a Rich Representation: Visualizing spectrograms revealed significant linguistic differences between native and non-native speech, not just at the prosodic level but also at the segmental level. This observation was the primary motivation for using image-based GANs.
  • GANs for Segmental Correction: Unlike previous methods limited to prosodic features, GANs offer the potential to learn and correct segmental errors (individual sound pronunciations) by transforming spectrograms.
  • Cycle Consistency for Robustness: The inclusion of cycle consistency loss is crucial. It helps to constrain the vast possibilities of image-to-image translation, ensuring that the generated feedback is not only phonetically plausible but also retains the essential structure of the original utterance, making it "self-imitative."
  • Self-Imitation as a Learning Strategy: The core idea is that by transforming a learner's own voice into a more native-like version, the learner can more easily identify their errors and imitate the corrected sound, leading to more effective learning.

Key Results

  • Qualitative Spectrogram Improvement: Figure 3 shows that the GAN-based generator, particularly the CycleGAN, progressively learns to transform non-native spectrograms to closely resemble native spectrograms, even correcting errors like syllable deletion.
  • Perceptual Evaluation (MOS Scores): A perceptual test with human raters showed that the proposed CycleGAN-based method achieved significantly higher Mean Opinion Scores (MOS) across various criteria (holistic impression, segmental correction, suprasegmental correction, sound quality, and speaker voice imitability) compared to the baseline PSOLA method and the Pix2Pix-only GAN.
  • CycleGAN achieved an average score of 3.776, while PSOLA scored 3.259.
  • The CycleGAN method consistently scored 3 or above in all categories, indicating no degradation in sound quality and effective correction.
  • Limitations of Baseline: The PSOLA baseline was reported to often produce trivial corrections or no corrections at all. The Pix2Pix-only GAN often failed to make corrections.

Limitations and Future Work

  • Sound Quality Loss: The authors acknowledge that the Griffin-Lim inversion process for converting spectrograms back to audio might introduce some loss in sound quality.
  • Imitability Score: While CycleGAN performed well overall, its imitability score was lower than PSOLA. Future work could focus on improving speaker voice imitability.
  • Real-time Interaction: The current system is not real-time. Testing the feasibility of real-time interactive response generation using GAN algorithms is a key future direction.
  • Longer Audio Recordings: Extending the approach to handle longer audio recordings and explore more sophisticated conditioning strategies is also suggested.
  • Improved Inversion Method: Developing a better method for spectrogram-to-audio inversion is recommended to enhance sound quality.

Summary updated

BibTeX
@inproceedings{yang2019self,
  author = {Seung Hee Yang and Minhwa Chung},
  title = {{Self-Imitating Feedback Generation Using GAN for Computer-Assisted Pronunciation Training}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure