Speech-enhanced and Noise-aware Networks for Robust Speech Recognition

Hung-Shin Lee, Pin-Yuan Chen, Yao-Fei Cheng, Yu Tsao, Hsin-Min Wang

TL;DR

This paper introduces a robust Automatic Speech Recognition (ASR) framework that combines speech enhancement, noise-aware training, and multi-condition training into a single jointly optimized network. By decomposing noisy speech into distinct clean speech and noise features, and optimizing them alongside sequence-level ASR objectives (LF-MMI), the proposed system achieves state-of-the-art performance on challenging noisy benchmarks.

What Problem Does This Paper Address?

Real-world ASR systems frequently degrade when deployed in noisy environments or when there is a mismatch between the training and testing microphones (channel mismatch). Traditional solutions typically fall into three buckets:

  1. Speech Enhancement (SE): Pre-processing audio to remove noise, which often introduces artificial distortions that confuse the downstream ASR.
  2. Multi-Condition Training (MCT): Training the ASR on a mix of clean and noisy data, which helps but struggles with unexpected noise types.
  3. Joint Training (JT): Training the SE and ASR models together.

While joint training is promising, previous GMM/DNN hybrid systems only optimized frame-level cross-entropy (CE) losses. They missed out on sequence-level discriminative training objectives, which look at the entire utterance context and are crucial for modern ASR performance. This paper bridges that gap.

Methods

The authors propose a unified framework consisting of two cascaded neural modules trained via a joint loss function:

  1. Speech-Enhanced and Noise-Aware Network (SENAN): A multi-task autoencoder that takes noisy acoustic features (MFCCs + i-vectors) and decomposes them into two separate streams: predicted clean speech features and predicted noise features.
  2. Acoustic Model (AM): Implemented using either a Factorized Time Delay Neural Network (TDNN-F) or a CNN-TDNNF architecture. The input to the AM is a concatenation of three feature sets: the original noisy features, the enhanced speech features, and the noise-aware features.
The proposed speech-enhanced and noise-aware acoustic model architecture.

Key Innovations:

  • STAT Aggregation: To make the estimated noise feature useful, the authors pass it through an aggregation function. They discovered that computing the mean and variance over a long context window (150 frames) provides a stable "noise profile" that helps the acoustic model filter out background interference.
  • Unified Loss Function: The entire network is optimized simultaneously using a combination of frame-level cross-entropy, feature reconstruction error (MSE for both speech and noise), and utterance-level Lattice-Free Maximum Mutual Information (LF-MMI).

Datasets

The framework was evaluated on two widely recognized speech recognition datasets:

  • Aurora-4: A medium-vocabulary task based on the Wall Street Journal corpus, featuring synthetic additive noise and channel distortions across four evaluation subsets (A: clean, B: noisy, C: clean+channel, D: noisy+channel).
  • AMI Meeting Corpus: A highly challenging real-world dataset consisting of 77 hours of meeting recordings. The authors used individual headset microphones as the clean ground truth and a single distant microphone (SDM) to represent the noisy environment.

Key Insights and Ideas

  • Noise is a Feature, Not Just a Nuisance: Instead of simply trying to erase noise during pre-processing, explicitly isolating the noise vector and feeding it into the acoustic model allows the network to learn how to actively ignore it.
  • Sequence-Level Optimization Matters: Incorporating LF-MMI into the joint training loop ensures that the speech enhancement module alters the features in a way that directly benefits whole-word and sentence-level recognition, rather than just optimizing for pixel-perfect spectrogram reconstruction.
  • Statistical Context Wins for Noise: When comparing different ways to process the noise vector, long-term statistical averages (mean and variance) outperformed local frame context and self-attention mechanisms. Noise patterns are generally stationary over short periods, making statistical tracking highly effective.

Key Results

Aurora-4 Benchmark

The proposed CNN-TDNNF system combined with SpecAugment achieved a state-of-the-art Word Error Rate (WER) of 3.55%.

  • Compared to existing top-performing systems in the literature, the proposed method achieved a relative WER reduction of 15.20% using bigram language models and 33.53% using trigram language models during decoding.

AMI Real-World Benchmark

On the far more difficult AMI distant-microphone task, the framework successfully complemented standard data augmentation:

  • Baseline CNN-TDNNF: 37.0% WER
  • With SpecAugment: 35.9% WER
  • With SpecAugment + SENAN (Proposed): 35.5% WER

Limitations and Future Work

The most telling finding in the paper comes from their Oracle experiment. When the authors replaced the SENAN module's estimated clean speech and noise features with the actual ground-truth clean speech and noise, the Aurora-4 WER plummeted to an astonishing 1.61%.

This massive gap (3.55% vs 1.61%) reveals a clear limitation: the system's performance is heavily bottlenecked by the accuracy of the multi-task autoencoder. If the speech enhancement network can be improved to generate cleaner separations, massive ASR performance gains are still on the table. Future work will likely target more advanced generative or self-supervised architectures for the SENAN component.

Summary updated

Abstract

Compensation for channel mismatch and noise interference is essential for robust automatic speech recognition. Enhanced speech has been introduced into the multi-condition training of acoustic models to improve their generalization ability. In this paper, a noise-aware training framework based on two cascaded neural structures is proposed to jointly optimize speech enhancement and speech recognition. The feature enhancement module is composed of a multi-task autoencoder, where noisy speech is decomposed into clean speech and noise. By concatenating its enhanced, noise-aware, and noisy features for each frame, the acoustic-modeling module maps each feature-augmented frame into a triphone state by optimizing the lattice-free maximum mutual information and cross entropy between the predicted and actual state sequences. On top of the factorized time delay neural network (TDNN-F) and its convolutional variant (CNN-TDNNF), both with SpecAug, the two proposed systems achieve word error rate (WER) of 3.90% and 3.55%, respectively, on the Aurora-4 task. Compared with the best existing systems that use bigram and trigram language models for decoding, the proposed CNN-TDNNF-based system achieves a relative WER reduction of 15.20% and 33.53%, respectively. In addition, the proposed CNN-TDNNF-based system also outperforms the baseline CNN-TDNNF system on the AMI task.
BibTeX
@inproceedings{lee2022speech,
  author = {Hung-Shin Lee and Pin-Yuan Chen and Yao-Fei Cheng and Yu Tsao and Hsin-Min Wang},
  title = {{Speech-enhanced and Noise-aware Networks for Robust Speech Recognition}},
  booktitle = {2022},
  year = {2022},
}
Figure