Whisper Augmented End-to-End/Hybrid Speech Recognition System — CycleGAN Approach

Prithvi R.r. Gudepu, Gowtham P. Vadisetti, Abhishek Niranjan, Kinnera Saranu, Raghava Sarma, M. Ali Basha Shaik, Periyasamy ParamasivamInterspeech 2020

TL;DR

This paper tackles the challenge of poor speech recognition performance on whispered speech by introducing a novel data augmentation technique using CycleGAN to generate synthetic whisper data from normal speech. This approach significantly reduces Word Error Rate (WER) on whispered speech for both end-to-end and hybrid ASR systems, even improving performance on normal speech in some cases.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are ubiquitous in virtual assistants and other applications, but they often struggle with whispered speech. This is primarily due to the lack of large, annotated whisper corpora for training. Whispered speech differs acoustically from normal speech because the vocal folds are held open, leading to the absence of fundamental frequency (F0) and altered formant structures, which can confuse ASR models trained only on phonated speech. This limitation hinders the usability of ASR in scenarios requiring privacy or quiet interaction, such as in meetings or at home.

Methods

The core of this paper's contribution is the use of Cycle-Consistent Generative Adversarial Networks (CycleGAN) for data augmentation. CycleGAN is an unsupervised image-to-image translation framework that can learn mappings between two domains without paired data. In this context, it's adapted to translate speech features from the "normal speech" domain to the "whisper" domain and vice-versa.

The authors use a CycleGAN architecture with two generators ($G_{s2w}$ and $G_{w2s}$) and two discriminators ($D_s$ and $D_w$). $G_{s2w}$ learns to convert normal speech features to whisper features, while $G_{w2s}$ does the reverse. The "cycle-consistency" loss ensures that if you convert speech to whisper and then back to speech, you should ideally get the original speech back, and similarly for whisper. This helps stabilize training and generate more realistic outputs.

To align the parallel normal and whisper speech data used for training the CycleGAN, they employ Dynamic Time Warping (DTW). DTW is a standard algorithm for finding the optimal alignment between two temporal sequences that may vary in speed or timing.

For generating the actual audio from the synthesized whisper features, they use the WORLD vocoder, a well-established system for speech synthesis that estimates fundamental frequency, spectral envelope, and aperiodicity parameters. For synthetic whisper, the F0 is set to zero, and the aperiodic parameter is used to generate the sound.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Librispeech: A large corpus (960 hours) used for training baseline ASR systems and as a source for generating synthetic whisper data.
  • wTIMIT: A smaller, parallel corpus (26 hours) containing both normal and whispered speech with different accents, used for evaluation.
  • Internally Recorded Trainset (TR1): A parallel corpus of normal and whispered speech recorded internally, used for training the CycleGAN.
  • Internally Recorded Testset (TS1): Internal test data for near-field whispered and normal speech.
  • Internally Recorded Far-field Testset (TS2): Internal test data for far-field whispered and normal speech.

Key Insights and Ideas

  • CycleGAN for Speech Domain Translation: The paper demonstrates the effectiveness of CycleGAN, a technique primarily known for image translation, in the domain of speech feature conversion, specifically for generating synthetic whispered speech.
  • Addressing Data Scarcity: By generating synthetic whisper data from abundant normal speech, CycleGAN offers a practical solution to the critical bottleneck of limited whisper corpora in ASR research.
  • Improved ASR Performance Across Architectures: The augmentation technique proves beneficial for both state-of-the-art end-to-end (E2E) and hybrid ASR systems, suggesting its broad applicability.
  • Whisper-Specific Augmentation Benefits: The most significant gains are observed on whispered speech test sets, directly addressing the core problem.
  • Unexpected Gains on Normal Speech: Interestingly, the augmented models also show improvements or minimal degradation on normal speech test sets, indicating that the synthetic data might implicitly improve robustness.

Key Results

The paper reports significant reductions in Word Error Rate (WER):

  • wTIMIT Whisper: A 13.5% relative WER reduction for E2E ASR and 7.6% for Hybrid ASR when augmenting with synthetic whisper data.
  • Internally Recorded Near-field (TS1): A 23% WER reduction on clean whisper, 20% on noisy whisper, and 16% on normal speech test cases.
  • Internally Recorded Far-field (TS2): A 26% WER reduction on 5-meter test cases and 15% on 3-meter test cases for far-field normal speech.
  • wTIMIT (Normal Speech): Minimal increase in WER on normal test sets for both E2E and Hybrid models.

Limitations and Future Work

  • Scope for Improvement: While CycleGAN shows promise, the authors note that experiments with naturally recorded whisper data suggest there's still room for improvement in the quality of synthetic data generation.
  • Language Dependency: The authors observe that the CycleGAN approach might have less language dependency and plan to explore building unified models for multiple languages.
  • Open-Sourcing: The team plans to release their CycleGAN models as open-source in the near future, which would be a valuable contribution to the research community.
  • Further Augmentation: Models trained with additional real whisper data (wTIMIT and TR1) showed further improvements, suggesting that combining synthetic and real data is a powerful strategy.

Summary updated

BibTeX
@inproceedings{gudepu2020whisper,
  author = {Prithvi R.R. Gudepu and Gowtham P. Vadisetti and Abhishek Niranjan and Kinnera Saranu and Raghava Sarma and M. Ali Basha Shaik and Periyasamy Paramasivam},
  title = {{Whisper Augmented End-to-End/Hybrid Speech Recognition System — CycleGAN Approach}},
  booktitle = {Interspeech},
  year = {2020},
}
Figure