Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognition

Hiroshi Sato, Tsubasa Ochiai, Marc Delcroix, Keisuke Kinoshita, Naoyuki Kamo, Takafumi Moriya

TL;DR

This paper tackles the challenge of overlapping speech recognition by proposing a novel neural network-based switching mechanism. Instead of relying on heuristic rules based on signal-to-noise ratios, their approach learns to dynamically decide whether to use enhanced or original audio for better speech recognition, significantly improving accuracy and introducing a "soft-switching" method for further gains.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are increasingly capable, but they struggle with challenging acoustic conditions, particularly overlapping speech (when multiple people speak simultaneously) and background noise. A common approach to mitigate these issues is to use a Speech Enhancement (SE) front-end to clean up the audio before feeding it to the ASR back-end. However, SE systems, especially those based on deep neural networks (DNNs), can introduce "processing artifacts" that, paradoxically, can degrade ASR performance, even in regions where speech is enhanced.

Previous work attempted to address this by using rule-based switching: if certain signal quality metrics (like Signal-to-Interference Ratio or Signal-to-Noise Ratio) indicated poor enhancement quality, the system would revert to using the original, unenhanced audio. While this showed promise, the rules were heuristic and not always optimal, as signal quality alone doesn't perfectly predict ASR performance. This paper aims to overcome these limitations by learning a more robust and optimal switching strategy directly from data.

Methods

The core innovation is a DNN-based switching model. This model takes both the original observed mixture (Y) and the enhanced speech (Ŝ) as input and outputs probabilities indicating whether the ASR performance would be better with the observed mixture (p₀) or the enhanced speech (p₁). This is framed as a binary classification task.

To train this switching model, the authors generate ground truth labels by comparing the ASR performance (measured by Character Error Rate - CER) on the original mixture versus the enhanced speech for each utterance. If the original mixture yields a lower CER, the label is [1, 0]; otherwise, it's [0, 1]. The switching model's parameters are then optimized using a binary cross-entropy loss.

A key extension is soft-switching. Instead of making a hard decision (use Y or Ŝ), soft-switching uses the continuous output probabilities (p₀, p₁) from the switching model to compute a weighted sum of the observed mixture and the enhanced speech: Ŝ_weighted = p₀ * Y + (1 - p₀) * Ŝ. This allows the system to leverage the uncertainty in the switching model's prediction and potentially achieve smoother, more robust performance.

Datasets

The experiments were conducted using simulated mixtures generated from:

  • Corpus of Spontaneous Japanese (CSJ) for speech recordings.
  • CHIME-4 corpus for noise recordings.

The authors detail several dataset splits for training the SE model, the ASR model, and the switching model, ensuring that the switching model is trained on data that is aware of the SE model's behaviors on unseen data. The evaluation dataset consisted of almost fully overlapping speech (approximately 94% overlap ratio).

Key Insights and Ideas

  • Signal quality metrics (SIR/SNR) are insufficient: The paper empirically demonstrates that even with perfect SIR/SNR estimation, rule-based switching has performance limitations. This is because the relationship between signal quality and ASR performance is not always straightforward and can vary significantly even within similar SIR/SNR conditions.
  • Learning to switch is superior: A DNN that directly learns to predict ASR performance based on the enhanced and observed signals outperforms heuristic rule-based switching, even when compared to an oracle rule-based switcher.
  • Soft-switching offers further gains: By leveraging the probabilistic output of the switching model, soft-switching (a weighted combination of enhanced and observed signals) achieves even better ASR performance than hard-switching, indicating that incorporating uncertainty is beneficial.
  • Flexibility for real-world applications: The proposed switching method can be trained without requiring paired clean sources and mixtures, making it adaptable to real-world scenarios where such data is scarce.

Key Results

  • The proposed learning-based hard-switching achieved comparable performance to the upper bound of conventional rule-based switching and outperformed it in several conditions.
  • Soft-switching with the best common weight achieved the best ASR performance, showing significant improvements.
  • Specifically, soft-switching achieved up to a 23% relative reduction in Character Error Rate (CER) compared to the conventional rule-based method.
  • The classification accuracy of the proposed switching model was 83.0%, higher than the 80.4% achieved by the conventional method.

Limitations and Future Work

  • Oracle Soft-Switching Gap: While the proposed soft-switching performed very well, there was still a gap between its performance and that of an "oracle soft-switching" where the optimal soft weight was searched for each individual utterance. This suggests that directly learning the optimal soft weight could further improve performance.
  • Real-world Data Adaptation: The current work primarily uses simulated data. The authors plan to apply the switching method to more realistic, real-recorded mixtures and investigate its performance on partially overlapping speech.
  • Joint Optimization: The paper mentions that direct learning of soft-switching by using the oracle soft weight could improve performance, hinting at potential future work in optimizing the soft-switching directly.

Summary updated

Abstract

The combination of a deep neural network (DNN) -based speech enhancement (SE) front-end and an automatic speech recognition (ASR) back-end is a widely used approach to implement overlapping speech recognition. However, the SE front-end generates processing artifacts that can degrade the ASR performance. We previously found that such performance degradation can occur even under fully overlapping conditions, depending on the signal-to-interference ratio (SIR) and signal-to-noise ratio (SNR). To mitigate the degradation, we introduced a rule-based method to switch the ASR input between the enhanced and observed signals, which showed promising results. However, the rule's optimality was unclear because it was heuristically designed and based only on SIR and SNR values. In this work, we propose a DNN-based switching method that directly estimates whether ASR will perform better on the enhanced or observed signals. We also introduce soft-switching that computes a weighted sum of the enhanced and observed signals for ASR input, with weights given by the switching model's output posteriors. The proposed learning-based switching showed performance comparable to that of rule-based oracle switching. The soft-switching further improved the ASR performance and achieved a relative character error rate reduction of up to 23 % as compared with the conventional method.
BibTeX
@inproceedings{sato2022learning,
  author = {Hiroshi Sato and Tsubasa Ochiai and Marc Delcroix and Keisuke Kinoshita and Naoyuki Kamo and Takafumi Moriya},
  title = {{Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746347},
}
Figure