Multi-Task Multi-Network Joint-Learning of Deep Residual Networks and Cycle-Consistency Generative Adversarial Networks for Robust Speech Recognition

Shengkui Zhao, Chongjia Ni, Rong Tong, Bin MaInterspeech 2019

TL;DR

This paper tackles the challenge of making speech recognition systems robust to noisy environments. They introduce a novel multi-task, multi-network learning framework that combines the power of deep residual networks (ResNets) with cycle-consistent generative adversarial networks (CycleGANs) to improve both speech enhancement and acoustic modeling simultaneously, leading to significantly lower word error rates.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle in real-world scenarios due to background noise and reverberations, leading to a significant drop in accuracy compared to clean, close-talking conditions. While various approaches like speech enhancement and robust acoustic modeling have been explored, they often have limitations. Speech enhancement techniques can sometimes distort the target speech, and traditional robust acoustic modeling methods can degrade when training and testing noise conditions mismatch. Multi-task joint-learning schemes have shown promise by unifying feature enhancement and acoustic modeling, but they typically require paired clean and noisy data. Generative Adversarial Networks (GANs) offered a way to bypass this paired data requirement, but regular GANs are prone to training instability (mode collapse), hindering their effectiveness for speech. This paper aims to overcome these limitations by proposing a more stable and effective GAN-based joint-learning framework for robust ASR.

Methods

The core of this paper is a Multi-Task Multi-Network Joint-Learning Framework. This framework integrates several key components:

  • Cycle-Consistent Generative Adversarial Networks (CycleGANs): Unlike standard GANs, CycleGANs learn mappings between domains (e.g., noisy speech to clean speech) without requiring paired data. They achieve this by enforcing a "cycle consistency loss," meaning that if you translate data from domain X to Y and then back to X, you should get something close to the original. This makes the training process more stable and less prone to mode collapse, a common issue with regular GANs.
  • Deep Residual Networks (ResNets): ResNets are used to build deeper neural networks by incorporating "shortcut connections." These connections help mitigate the vanishing gradient problem, allowing for the training of much deeper and more powerful models, which are crucial for capturing complex speech patterns.
  • Multi-Network Architecture with Ensemble Features: The proposed system uses a multi-network approach. It combines the output of a CycleGAN generator (which enhances noisy features) with the output of a parallel ResNet (which learns from noisy features directly). These features are then processed by a Squeeze-and-Excitation (SE) block, which adaptively recalibrates channel-wise feature responses, emphasizing important features and suppressing less useful ones. This ensemble of features is then fed into a senone label classifier.
  • Joint Optimization: The entire system is trained end-to-end by jointly optimizing multiple objectives:
  • Adversarial Loss: For the GAN component, to make enhanced features indistinguishable from clean features.
  • Cycle Consistency Loss: To ensure the integrity of the translation process.
  • Classification Loss: For the senone classifier to accurately predict speech units.

Datasets

  • CHiME-4 Dataset: This dataset was used for evaluation. It consists of real and simulated recordings of speech from the Wall Street Journal (WSJO) corpus, featuring various noise environments (bus, cafe, pedestrian area, street junction) and multiple microphone channels. The paper specifically mentions using the official one-channel track lists for development and evaluation.
  • Original WSJO training data: Used for generating real samples for GAN training.

Key Insights and Ideas

  • CycleGANs for Speech Stability: The paper highlights that regular GANs struggle with speech feature mapping due to its under-constrained nature, leading to mode collapse. CycleGANs, with their cycle consistency loss, provide a more stable and effective way to perform this mapping, preserving crucial speech characteristics.
  • Ensemble of Enhanced and Direct Features: Combining features enhanced by a CycleGAN with features learned directly by a parallel ResNet (and further refined by an SE block) leads to a richer and more discriminative representation for the acoustic model.
  • Deep Networks are Key: The use of deep ResNets in both the GAN components and the parallel network is crucial for achieving state-of-the-art performance, demonstrating the continued importance of network depth in ASR.
  • Joint Learning for Synergy: Unifying speech enhancement (via CycleGAN) and acoustic modeling (via ResNet and classifier) within a single, jointly optimized framework allows these tasks to benefit from each other, leading to superior robustness.

Key Results

The proposed "Dual-CycleGAN-ResNet×33" approach achieved the best performance among all compared systems on the CHiME-4 dataset. Specifically:

  • Significant WER Reduction: The paper reports substantial improvements in Word Error Rates (WERs) compared to state-of-the-art joint-learning approaches. For instance, on the test set, their model achieved a WER of 25.74% (average of simulated and real), outperforming previous methods.
  • Qualitative Improvement: Figure 2 visually demonstrates the effectiveness of their CycleGAN approach. It shows that the enhanced features preserve speech structures much better than those generated by a regular GAN approach, which often fails to maintain these characteristics.

Limitations and Future Work

  • Hyper-parameter Tuning: The paper mentions that hyper-parameters $\alpha$ and $\beta$ (weighting the GAN and cycle consistency losses) were tuned for best performance, implying that careful tuning is necessary.
  • Computational Cost: While not explicitly detailed as a limitation, the use of multiple deep networks and GANs suggests a potentially higher computational cost for training and inference compared to simpler models.
  • Future Work: The paper implicitly suggests that further exploration of multi-network architectures and advanced GAN variants could lead to even greater improvements in robust ASR. The success of the SE block also points towards exploring other attention or recalibration mechanisms.

Summary updated

BibTeX
@inproceedings{zhao2019multi,
  author = {Shengkui Zhao and Chongjia Ni and Rong Tong and Bin Ma},
  title = {{Multi-Task Multi-Network Joint-Learning of Deep Residual Networks and Cycle-Consistency Generative Adversarial Networks for Robust Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure