Speech Audio Super-Resolution for Speech Recognition

Xinyu Li, Venkata Chebiyyam, Katrin KirchhoffInterspeech 2019 · DOI

Abstract

Researchers at Amazon AI have developed an end-to-end deep learning system that "up-rezzes" low-quality, narrow-band speech audio into high-quality, wide-band audio specifically to improve the performance of speech recognition (ASR) systems. By using a conditional Generative Adversarial Network (cGAN) equipped with task-specific loss functions—including Mel-Frequency Cepstral Coefficient (MFCC) comparisons—the system achieved a ~20% relative reduction in Word Error Rate (WER) compared to traditional methods.

Introduction

Most modern speech recognition models are trained on high-quality, wide-band (WB) audio, typically sampled at 16kHz. However, real-world audio often originates from narrow-band (NB) sources, such as legacy telephony (8kHz) or low-bandwidth streaming services. When this "muffled" audio is processed by a wide-band ASR model, performance degrades significantly.

While traditional "Bandwidth Extension" (BWE) techniques aim to restore missing high frequencies, they are primarily designed for human perceptual quality. This research addresses the gap where audio that sounds "good" to a person might still be "unreadable" to a machine. The authors focus on optimizing BWE specifically for downstream ASR accuracy, treating the problem as a signal restoration task where the narrow-band signal is viewed as a wide-band signal mixed with noise that cancels high frequencies.

Proposed Architecture

The core of the system is a conditional Generative Adversarial Network (cGAN). The architecture is designed to prioritize features relevant to ASR systems rather than just raw waveform similarity.

The proposed cGAN architecture featuring a U-Net generator and a discriminator with an MFCC extraction layer.

Generator and Discriminator Design

The generator employs a convolutional encoder-decoder structure with U-Net connections. These skip connections pass information directly from the input layers to the output layers, ensuring that the original low-frequency content remains untouched while the high-frequency components are synthesized.

Unlike standard GANs that operate on raw waveforms, the discriminator incorporates an MFCC extraction layer. By converting audio into a 2D Mel-Frequency Cepstral Coefficient representation before classification, the discriminator forces the generator to produce audio that contains the specific spectral "fingerprints" used by ASR models to understand speech.

Multi-Part Loss Functions

The model's training is guided by a composite loss function designed to balance physical accuracy with ASR utility:

  1. Adversarial Loss: The standard GAN objective to ensure realistic output.
  2. Perceptual Loss: Measures the distance between internal representations of a pre-trained ASR model when fed real vs. generated audio.
  3. MFCC Loss: A proxy for perceptual loss that calculates the distance between MFCC features, providing a computationally cheaper alternative to using a full ASR model.
  4. Mean Absolute Error (MAE): Ensures the generated waveform remains physically close to the original, implicitly handling phase loss in low-frequency components.

Experimental Results

The system was evaluated on the Speech Commands dataset (65,000 utterances) and the LibriSpeech corpus (~1000 hours). The cGAN approach significantly outperformed baseline methods like simple interpolation and previous deep-learning BWE models.

Method Speech Commands (WER) LibriSpeech (WER)
Interpolation (Baseline) 8.00% 12.66%
Previous GAN Method 7.15% 12.34%
Our Approach (with Perceptual Loss) 5.80% 12.17%
Real 16kHz Audio (Oracle) 4.54% 10.81%

Qualitatively, the cGAN is robust enough to fill the "spectral gap" often left by traditional low-pass filters around the 4kHz mark, without requiring prior knowledge of the specific cutoff frequency used during downsampling.

Spectrogram comparison of various bandwidth extension methods against the 16kHz ground truth.

Conclusion

This research demonstrates that machines and humans "hear" differently; optimizing for traditional signal-to-noise ratios does not guarantee better ASR performance. By incorporating ASR-specific features like MFCCs into the training loop, the model prioritizes the information most critical for machine understanding. Future work may involve extending this framework to joint noise cancellation and bandwidth extension, or exploring its application in cross-language scenarios.

Summary updated

BibTeX
@inproceedings{li2019speech,
  author = {Xinyu Li and Venkata Chebiyyam and Katrin Kirchhoff},
  title = {{Speech Audio Super-Resolution for Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-3043},
}
Figure