A framework for the evaluation of microscopic intelligibility models

Ricard Marxer, Martin Cooke, Jon BarkerInterspeech 2015 · OpenReview

Introduction

Traditional speech intelligibility models, such as the Articulation Index (AI) or the Speech Intelligibility Index (SII), are primarily "macroscopic." They excel at predicting aggregate statistics—specifically, the average percentage of words a group of listeners will correctly identify in a given noisy environment. However, these models lack the granularity to explain specific human errors. They cannot predict why a listener might hear "ministro" when the target word was "maestro."

"Microscopic" models aim to fill this gap by predicting fine-grained responses, including phonetic confusion matrices or specific lexical misidentifications. Despite their potential, the field has lacked a standardized framework for evaluating these models. Researchers often rely on disparate datasets, closed-set tasks (e.g., choosing from a small list of consonants), or inconsistent metrics, making cross-model comparison nearly impossible. This paper introduces a rigorous, standardized framework to evaluate microscopic models using open-vocabulary, multisyllabic speech data.

The Evaluation Framework

The proposed framework evaluates models across three increasingly complex tasks, moving from simple error frequency to full word prediction:

  1. Confusion Frequency Prediction ($L^{freq}$): Predicting the probability that a phonetic error (substitution, deletion, or insertion) will occur at a specific position within a word.
  2. Confusion Characterization Prediction ($L^{char}$): Predicting the probability distribution of the type of confusion (e.g., which specific phoneme replaces the target).
  3. Full Word Confusion Prediction ($L^{full}$): Predicting the probability distribution of the entire word response provided by the listener.

Phonetic Alignment and Indexing

To enable sublexical analysis, the framework decomposes words into a sequence of phonemes and inter-phoneme spaces. This indexing allows the model to account for insertions that might occur between existing sounds.

Phonetic indexing of the target word 'maestro', including inter-phoneme spaces (indices 0, 2, 4, etc.) to allow for insertions.

A core challenge in microscopic modeling is that a listener's response may have a different number of phonemes than the target. The framework addresses this through phonetic alignment using "edit scripts" (match, substitution, deletion, insertion). Because a single response can often be aligned with a target in multiple ways with the same minimum edit cost, the authors implement a "benefit of the doubt" approach.

Example of multiple possible phonetic alignments between the target word 'maestro' and the misperceived response 'ministro'.

As shown in the figure above, the framework selects the alignment that yields the highest score for the model being tested. This prevents models from being penalized for the inherent ambiguity in how human speech perception is mapped to phonetic strings.

Experimental Validation

The framework was demonstrated using a large-scale Spanish confusions corpus containing 3,235 stimuli. These stimuli were 1-3 syllable words presented in various noise backgrounds. A unique feature of this corpus is its focus on "consistent misperceptions," where at least 6 out of 15+ listeners reported the exact same incorrect word.

To establish performance bounds, the authors introduced three "pseudo-models":

  • Oracle: Uses actual listener data to make predictions (the theoretical upper bound).
  • Statistical: Uses aggregate dataset statistics (a baseline representing macroscopic knowledge).
  • Random: Makes uniform, non-informative guesses (the lower bound).

Results and Discussion

Performance was measured using log-likelihood, where values closer to zero indicate better predictive accuracy.

Model $L^{freq}$ $L^{char}$ $L^{full}$
Oracle -0.645 -1.122 -4.689
Statistical -3.747 -10.952 -172.463
Random -7.631 -128.821 -184.831

The results reveal a massive performance gap between the Statistical baseline and the Oracle model, particularly in the Full Word task ($L^{full}$). This gap represents the "room for discovery" where true microscopic models can demonstrate value by capturing specific acoustic-phonetic nuances that simple frequency statistics overlook. The extreme difficulty of the $L^{full}$ task justifies the use of the simpler sub-tasks ($L^{freq}$ and $L^{char}$) to help researchers iteratively improve their models.

Conclusion

By providing a standardized metric and a sublexical alignment strategy, this framework allows for the objective comparison of microscopic intelligibility models. To encourage community adoption, the authors have launched an online evaluation service where researchers can upload model predictions and receive standardized performance scores against the Spanish confusions corpus.

Summary updated

BibTeX
@inproceedings{marxer2015framework,
  author = {Ricard Marxer and Martin Cooke and Jon Barker},
  title = {{A framework for the evaluation of microscopic intelligibility models}},
  booktitle = {Interspeech},
  year = {2015},
  url = {https://openreview.net/forum?id=BJ7QlrxO-z},
}
Figure