Improving Character Error Rate is Not Equal to Having Clean Speech: Speech Enhancement for ASR Systems with Black-Box Acoustic Models

Ryosuke Sawata, Yosuke Kashiwagi, Shusuke Takahashi

TL;DR

This paper introduces a novel training scheme for speech enhancement (SE) models that directly optimizes for Automatic Speech Recognition (ASR) performance, measured by Character Error Rate (CER). By using a differentiable "CER estimator" that mimics a black-box ASR system, they enable SE models to be trained to minimize CER without needing access to the ASR's internal workings, leading to significant improvements in ASR accuracy.

What Problem Does This Paper Address?

Traditional speech enhancement (SE) aims to improve the perceptual quality of speech, often by minimizing the difference between the enhanced and clean speech signals (e.g., using Mean Squared Error). However, this doesn't always translate to better performance for downstream Automatic Speech Recognition (ASR) systems. In fact, SE can sometimes degrade ASR accuracy. This degradation occurs because SE models can introduce distortions that create a mismatch with the acoustic models (AMs) within ASR systems, which are typically trained on clean speech. While methods like re-training or joint training the AM with SE exist, they are computationally expensive and impractical when the ASR system is a black-box (e.g., provided by a third party). This paper addresses the challenge of optimizing SE models for ASR performance (specifically CER) when the ASR system is a black-box and cannot be modified or retrained.

Methods

The core innovation is a "CER-centric" training scheme for SE models. This scheme involves two key components trained alternately:

  1. Speech Enhancement (SE) DNN: This is the standard SE model responsible for cleaning the noisy speech.
  2. CER Estimator DNN: This novel component is trained to mimic the Character Error Rate (CER) output of a black-box ASR system. It takes the output of the SE model and estimates the CER that the ASR would produce.

The training process works as follows:

  • The CER estimator is trained to minimize the difference between its estimated CER and the actual CER calculated by feeding the SE output into the black-box ASR. This makes the CER estimation process differentiable.
  • Once the CER estimator is trained, the SE model is then optimized to minimize the CER predicted by this estimator.

This approach allows the SE model to learn to produce speech that is not just perceptually clean, but specifically optimized for the ASR's understanding, even if the ASR's internal mechanisms are unknown. The authors also introduce specific data preparation techniques:

  • Normalization: They normalize both input and ground-truth spectrograms to have unit standard deviation (not necessarily zero mean) to mitigate power bias across frequency bands, making the CER estimator more robust to different ASR systems.
  • Data Augmentation: They employ SpecAugmentation (time and frequency masking) to encourage the SE model to learn which parts of the spectrogram are crucial for the ASR, further improving CER.

Datasets

  • Libri-light [24]: Used for simulating noisy speech for training and evaluation. A small part (9 hours) was used for training, and a separate part (1 hour) for inference and evaluation.
  • MUSAN [26]: Used as additive noise during training.
  • DEMAND [27]: Used as additive noise during inference and evaluation.

The paper simulates noisy speech by mixing clean speech from Libri-light with noise from MUSAN (training) or DEMAND (inference/evaluation) at varying Signal-to-Noise Ratios (SNRs).

Key Insights and Ideas

  • Clean Speech ≠ Good for ASR: The paper strongly demonstrates that improving perceptual quality metrics (like PESQ, STOI) does not automatically lead to better ASR performance (lower CER). In fact, aggressive noise suppression can sometimes hurt CER.
  • Differentiable CER Mimicry is Key: The core breakthrough is creating a differentiable proxy for the non-differentiable CER metric. This allows gradient-based optimization of SE models for ASR performance.
  • Black-Box Compatibility: The method is designed to work with any ASR system, regardless of its internal architecture or whether it's trainable, making it highly practical.
  • Data Preparation Matters: Specific normalization and augmentation strategies are shown to be crucial for the success of the CER-centric approach, improving robustness and effectiveness.

Key Results

  • The proposed CER-centric SE model achieved a relative CER improvement of 8.8% compared to a baseline MetricGAN trained with CER as the target metric.
  • An ablation study showed that adding their proposed normalization and SpecAugmentation techniques sequentially improved CER scores.
  • The CER-centric SE model achieved a significantly lower CER than both the input noisy speech and a conventional SE model (trained with MSE), even though the conventional SE model showed better perceptual quality metrics.
  • Spectrograms of the CER-centric SE output (Figure 2d) show more residual noise compared to conventional SE (Figure 2c), highlighting the trade-off between perceptual quality and ASR performance.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their method improves CER, other perceptual quality metrics (like CSIG, CBAL, COVL, PESQ, STOI, SegSNR) might be slightly inferior compared to a conventional SE model trained with MSE. This indicates a trade-off between optimizing for CER and perceptual quality.
  • The effectiveness of the method might depend on the specific black-box ASR system used.
  • Future Work:
    • The authors suggest that this CER-centric training scheme can be applied to other ASR performance metrics (e.g., Word Error Rate - WER).
  • Further exploration of data augmentation and normalization strategies could potentially yield even better results.
  • Investigating the impact of different black-box ASR architectures on the proposed SE method would be valuable.

Summary updated

Abstract

A deep neural network (DNN)-based speech enhancement (SE) aiming to maximize the performance of an automatic speech recognition (ASR) system is proposed in this paper. In order to optimize the DNN-based SE model in terms of the character error rate (CER), which is one of the metric to evaluate the ASR system and generally non-differentiable, our method uses two DNNs: one for speech processing and one for mimicking the output CERs derived through an acoustic model (AM). Then both of DNNs are alternately optimized in the training phase. Even if the AM is a black-box, e.g., like one provided by a third-party, the proposed method enables the DNN-based SE model to be optimized in terms of the CER since the DNN mimicking the AM is differentiable. Consequently, it becomes feasible to build CER-centric SE model that has no negative effect, e.g., additional calculation cost and changing network architecture, on the inference phase since our method is merely a training scheme for the existing DNN-based methods. Experimental results show that our method improved CER by 8.8% relative derived through a black-box AM although certain noise levels are kept.
BibTeX
@inproceedings{sawata2022improving,
  author = {Ryosuke Sawata and Yosuke Kashiwagi and Shusuke Takahashi},
  title = {{Improving Character Error Rate is Not Equal to Having Clean Speech: Speech Enhancement for ASR Systems with Black-Box Acoustic Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746398},
}
Figure