Validation of the Non-Intrusive Codebook-Based Short Time Objective Intelligibility Metric for Processed Speech

Charlotte Sørensen, Jesper B. Boldt, Mads G. ChristensenInterspeech 2019

TL;DR

This paper validates a novel non-intrusive speech intelligibility metric, NIC-STOI, which estimates clean speech from degraded signals using codebooks. The metric shows high correlation with subjective intelligibility and outperforms existing non-intrusive methods, particularly for speech processed with binary masks.

What Problem Does This Paper Address?

Predicting how understandable speech will be is crucial for developing and optimizing speech enhancement technologies, especially in hearing aids. Many existing objective intelligibility metrics, like STOI, require a clean "ground truth" reference signal. This is often impractical in real-world scenarios, such as live audio processing or when dealing with recordings where the original clean speech is unavailable. This limitation necessitates the development of "non-intrusive" metrics that can estimate intelligibility directly from the degraded signal. While some non-intrusive methods exist, they often struggle with complex distortions like non-linear processing, which is common in modern hearing aid algorithms.

Methods

The core of this paper is the validation of the Non-Intrusive Codebook-based Short Time Objective Intelligibility (NIC-STOI) metric. NIC-STOI builds upon the established intrusive STOI metric but replaces the need for a clean reference. It achieves this by:

  1. Estimating Reference Signal Statistics: NIC-STOI models the degraded speech signal as a combination of speech and noise. It uses pre-trained codebooks of speech and noise spectral characteristics, along with auto-regressive (AR) parameters, to estimate the spectral characteristics of the original clean speech and the noise. This estimation is performed by minimizing the Itakura-Saito divergence between the observed noisy spectrum and the modeled spectrum.
  2. TF Composition: Using the estimated speech and noise parameters, the time-frequency (TF) spectra of the estimated clean speech and noise are computed.
  3. Intelligibility Prediction: These estimated TF spectra are then fed into the standard STOI algorithm as if they were the actual clean and noisy signals, yielding a non-intrusive intelligibility score.
  4. Handling Binary Masks: A specific adjustment is made to account for non-linear processing like Ideal Binary Masks (IBMs). This involves a thresholding mechanism in the Itakura-Saito divergence calculation to disregard time-frequency units with very low energy, which are likely rendered inaudible by the masking process.

Datasets

  • Training Data:
    • Clean speech: 50 sentences from the Dantale II dataset (not used in testing).
  • Noise: 50 sentences of each noise type (bottling factory hall, cafe, Speech Shaped Noise (SSN), car noise) without IBM processing.
  • Evaluation Data:
    • Dantale II sentence material mixed with four noise types (bottling factory hall, cafe, SSN, car noise) at three different Signal-to-Noise Ratios (SNRs): 50% SRT, 20% SRT, and -60 dB.
  • The noisy signals were processed with Ideal Binary Masks (IBMs) at eight different Relative Criterion (RC) values.
  • Data from 5 subjects was used for training codebooks, and data from the remaining 10 subjects was used for testing.

Key Insights and Ideas

  • Bridging the Intrusive-Non-Intrusive Gap: NIC-STOI effectively bridges the gap between intrusive and non-intrusive intelligibility metrics by cleverly estimating the necessary clean reference signal statistics.
  • Robustness to Non-linear Processing: The metric demonstrates significant promise in handling non-linear distortions like Ideal Binary Masks, a critical capability for evaluating modern speech enhancement systems.
  • Codebook Power: The use of pre-trained codebooks for speech and noise spectral characteristics is a key enabler for estimating the clean signal's properties without direct access.
  • Performance Close to Intrusive Metrics: The validation shows that NIC-STOI achieves intelligibility prediction accuracy remarkably close to the intrusive STOI metric, even for complex processing.

Key Results

  • NIC-STOI shows a high correlation with subjective intelligibility scores across various noise conditions and SNRs, particularly for speech processed with IBMs.
  • The metric is highly correlated with the original intrusive STOI metric, validating its approach.
  • When compared to other non-intrusive metrics (NI-STOI, SRMR-norm, SRMR), NIC-STOI outperforms them significantly in terms of Pearson's correlation (ρ), Kendall's tau (τ), and RMSE. For instance, NIC-STOI achieved a ρ of 0.940 and RMSE of 11.4%, while the next best non-intrusive metric (NI-STOI) had ρ=0.711 and RMSE=25.2%.
  • The performance of NIC-STOI is noted to be slightly lower for the SSN noise condition, potentially due to how the noise codebook was trained.

Limitations and Future Work

  • Noise Codebook Weighting: The authors suggest that the slightly lower performance on SSN noise might be explained by the noise codebook weighting this condition less during training. Future work could explore more balanced noise codebook training.
  • Comparison with NI-STOI: While NIC-STOI outperforms NI-STOI, the paper notes that NI-STOI's performance is still competitive, especially when the cafe noise condition is excluded from the logistic mapping. Further investigation into the specific strengths and weaknesses of each metric in different scenarios could be beneficial.
  • Generalizability: While validated on a diverse dataset, further testing on even broader ranges of noise types, speech distortions, and processing algorithms would strengthen the metric's generalizability.

Summary updated

BibTeX
@inproceedings{srensen2019validation,
  author = {Charlotte Sørensen and Jesper B. Boldt and Mads G. Christensen},
  title = {{Validation of the Non-Intrusive Codebook-Based Short Time Objective Intelligibility Metric for Processed Speech}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure