An instrumental intelligibility metric based on information theory

Steven Van Kuyk, W. Bastiaan Kleijn, Richard C. Hendriks

Abstract

This paper introduces Speech Intelligibility in Bits (SIIB), a novel instrumental metric designed to predict human speech intelligibility by calculating the mutual information rate between a talker and a listener. Unlike traditional heuristic-based metrics, SIIB accounts for temporal dependencies and talker variability. Evaluation across diverse datasets demonstrates that SIIB significantly outperforms existing standards like STOI, particularly in predicting the performance of modern, non-linear speech enhancement algorithms.

Introduction

In speech processing, evaluating the effectiveness of algorithms—such as noise suppression or speech enhancement—is critical. While human listening tests remain the gold standard, they are resource-intensive and expensive. Consequently, researchers rely on "instrumental metrics"—mathematical formulas that predict human scores. However, current metrics often suffer from two primary flaws: a reliance on heuristic observations (e.g., STOI) that fail under non-linear processing, and an oversimplification of speech as a memoryless signal with independent frequency bands.

Methodology

The authors propose modeling speech communication as a Markov chain: Message → Clean Speech → Degraded Speech. The core of the SIIB metric is the estimation of the Mutual Information (MI) rate in bits per second. To ensure accuracy and tractability, the metric incorporates several key innovations:

  • Auditory Modeling: Signals are processed through a Gammatone filterbank and temporal masking to simulate human hearing.
  • Karhunen-Loève Transform (KLT): This is used to decorrelate speech signals across time and frequency, allowing for the summation of information across independent components rather than overestimating it.
  • Talker Variability: A "production noise" factor is included to account for the natural variation in how different individuals pronounce the same message.
  • Non-parametric Estimation: A k-nearest neighbor (k-NN) estimator calculates MI directly from observed samples, avoiding restrictive assumptions about probability distributions.

Evaluation and Results

SIIB was evaluated against four diverse datasets: JensenSCNR (single-channel noise reduction), KleijnPRE (pre-processing enhancement), CookePRE (diverse enhancement algorithms), and KjemsITFS (binary masking).

Scatter plots comparing human intelligibility scores against various metrics across four datasets. SIIB (rightmost column) shows the most consistent alignment with human data across all conditions.

As shown in the scatter plots above, SIIB maintains a tight fit to the human intelligibility curve across all conditions. Quantitatively, SIIB achieved a mean Pearson correlation (ρ) of 0.95, notably higher than MIKNN (0.86), STOI (0.89), and ESTOI (0.92).

Table II: Pearson correlation coefficients (ρ) for various metrics. SIIB demonstrates the highest mean correlation and exceptional robustness, particularly in the challenging KleijnPRE condition.

A key finding is SIIB's robustness in the KleijnPRE dataset. While traditional metrics like STOI faltered when signal envelopes were modified by optimal energy redistribution, SIIB maintained a correlation of 0.98 by focusing on preserved information rather than signal shape. Furthermore, SIIB led in rank consistency, achieving a mean Kendall’s tau (τ) of 0.82.

Discussion and Limitations

SIIB represents a significant advancement by treating speech understanding as a problem of information transfer. It naturally accounts for the "glimpse effect"—the human ability to listen in the gaps of noise—without requiring special heuristics.

However, some limitations remain. SIIB performed slightly worse on the KjemsITFS dataset, likely because binary masking creates "unnatural" sounds that the human brain cannot decode efficiently despite the information being technically present. Additionally, SIIB estimates a maximum information rate of approximately 150 bits/s, which is higher than linguistic models suggest, indicating that some statistical dependencies remain. Finally, the use of k-NN estimation makes SIIB more computationally demanding than simpler correlation-based metrics.

Conclusion

By grounding intelligibility in information theory, SIIB provides a more robust and theoretically sound alternative to heuristic metrics. Its superior performance across varied noise conditions makes it a powerful tool for evaluating the next generation of speech enhancement technologies.

Summary updated

BibTeX
@article{kuyk2018instrumental,
  author = {Steven Van Kuyk and W. Bastiaan Kleijn and Richard C. Hendriks},
  title = {{An instrumental intelligibility metric based on information theory}},
  journal = {IEEE Signal Processing Letters},
  year = {2018},
  doi = {10.1109/lsp.2017.2774250},
}
Figure