A Cross-Entropy-Guided (CEG) Measure for Speech Enhancement Front-End Assessing Performances of Back-End Automatic Speech Recognition

Li Chai, Jun Du, Chin-Hui Lee

In the field of speech processing, speech enhancement (SE) is often employed as a front-end to improve the performance of back-end automatic speech recognition (ASR) systems in noisy environments. Traditionally, the effectiveness of SE algorithms is evaluated using instrumental measures like PESQ (Perceptual Evaluation of Speech Quality) or STOI (Short-Time Objective Intelligibility). However, these metrics were designed to model human auditory perception and often fail to correlate well with the Word Error Rate (WER) of ASR systems. This paper introduces the Cross-Entropy-Guided (CEG) measure, a novel metric specifically designed to predict ASR performance by leveraging the internal states of the ASR system itself.

The Need for ASR-Centric Metrics

The primary goal of speech enhancement for ASR is to reduce WER. Measuring WER directly is computationally expensive and requires manual transcriptions, making it impractical for rapid algorithm development or real-time optimization. While PESQ and STOI are convenient, they focus on speech quality and intelligibility for human listeners. A speech enhancement algorithm might improve these perceptual qualities while simultaneously introducing artifacts or distortions that degrade the ASR system's ability to recognize phonemes. This discrepancy creates a significant gap between "human-friendly" and "machine-friendly" audio.

The Cross-Entropy-Guided (CEG) Framework

The proposed CEG measure bridges this gap by utilizing the ASR system’s acoustic model to evaluate the impact of speech enhancement. Unlike traditional black-box metrics, CEG looks at the high-level internal representations generated by the ASR system.

The architecture of the proposed CEG measure calculation process.

As illustrated in the figure above, the CEG calculation involves three main stages:

  1. Feature Extraction: Both clean and enhanced speech signals are converted into low-level acoustic features (e.g., MFCCs or FBANK features).
  2. Nonlinear Mapping: These features are passed through the ASR system's acoustic model (typically a deep neural network). Instead of final text output, the measure extracts the state posterior probabilities—the network's internal confidence scores for various speech units (phonemes) at each time frame.
  3. Measure Calculation: The CEG is computed as the cross-entropy between the posterior probabilities of the clean speech and those of the enhanced speech. By comparing these internal "guesses," CEG quantifies how much the enhancement process has deviated from the ideal acoustic representation expected by the ASR back-end.

Experimental Evaluation

The researchers validated the CEG measure using the CHiME-4 benchmark, which includes diverse noise types (Cafe, Street, Bus, Pedestrian) and various signal-to-noise ratios (SNRs) from -5dB to 20dB. The evaluation spanned multiple acoustic model architectures (DNN, DNNsMBR, DCNN) and language models (3-gram, 5-gram, RNN) to ensure the metric's robustness across different ASR configurations.

Key Results and Insights

The experimental results demonstrate that CEG is a far more reliable predictor of ASR performance than traditional metrics.

Correlation between various instrumental measures (Entropy, PESQ, STOI, CEG) and Word Error Rate (WER) under different training conditions.

As shown in the scatter plots, CEG exhibits a significantly higher Pearson correlation with WER compared to PESQ and STOI. In multi-condition training modes, CEG achieved a correlation of approximately 0.74, whereas PESQ and STOI lagged behind at 0.55 and 0.63, respectively. This superior performance was consistent across different noise types and ASR back-end architectures.

A critical insight from this study is that CEG can identify instances of "over-processing." Some SE algorithms might yield high PESQ scores by aggressively removing noise, but in doing so, they "smear" the acoustic cues essential for the ASR model. CEG detects this as a mismatch in the internal probability distributions, correctly predicting a rise in WER where traditional metrics might predict an improvement.

Conclusion and Future Potential

The CEG measure provides a powerful tool for developing ASR-friendly speech enhancement front-ends. Its primary advantage lies in its ability to reflect the specific needs of the machine back-end rather than human perception. Furthermore, because CEG is based on a differentiable cross-entropy function, it holds significant potential as a loss function for training deep learning-based SE models directly to minimize ASR error.

Future work will involve extending the CEG framework to modern end-to-end (E2E) ASR architectures, where internal representations differ from the state posteriors used in hybrid ANN-HMM systems.

Summary updated

Abstract

One challenging problem of robust automatic speech recognition (ASR) is how to measure the goodness of a speech enhancement algorithm without calculating word error rate (WER) due to the high costs of manual transcriptions, language modeling and decoding process. In this study, a novel cross-entropy-guided (CEG) measure is proposed for assessing if enhanced speech predicted by a speech enhancement algorithm would produce a good performance for robust ASR. CEG consists of three consecutive steps, namely the low-level representations via the feature extraction, high-level representations via the nonlinear mapping with the acoustic model, and the final CEG calculation between the high-level representations of clean and enhanced speech. Specifically, state posterior probabilities from the output of the neural network for the acoustic model are adopted as the high-level representations and a cross-entropy criterion is used to calculate CEG. Experimental results show that CEG could consistently yield the highest correlations with WER and achieve the most accurate assessment of the ASR performance when compared to distortion measures based on human auditory perception and an acoustic confidence measure. Potentially, CEG could be adopted to guide the parameter optimization of deep learning based speech enhancement algorithms to further improve the ASR performance.
BibTeX
@inproceedings{chai2019cross,
  author = {Li Chai and Jun Du and Chin-Hui Lee},
  title = {{A Cross-Entropy-Guided (CEG) Measure for Speech Enhancement Front-End Assessing Performances of Back-End Automatic Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2511},
}
Figure