A Comparison of Sentence-Level Speech Intelligibility Metrics

Alexander Kain, Max Del Giudice, Kris TjadenInterspeech 2017

TL;DR

This paper explores novel, automated acoustic metrics to predict speech intelligibility, moving beyond traditional, labor-intensive methods. They find that measures of "separability" in feature spaces, particularly those derived from spectral features rather than just formant frequencies, show more promise for predicting intelligibility, especially in disordered speech.

What Problem Does This Paper Address?

Predicting how intelligible someone's speech is, especially for individuals with speech disorders like dysarthria, is crucial for diagnosis and treatment. Traditional methods often rely on manual analysis of acoustic features, like vowel formant frequencies, which is time-consuming and prone to errors, particularly with disordered speech. The authors aim to develop fully automated acoustic metrics that can accurately predict sentence-level speech intelligibility, hypothesizing that greater variability in speech features correlates with better intelligibility. They want to find metrics that are robust enough to work across different speaker groups, including those with mild speech impairments.

Methods

The paper investigates several approaches to measuring speech feature variability:

  • Vowel Space Area (VSA) based on Formant Frequencies: They first reproduce a standard VSA calculation using automatically extracted formant frequencies (F1/F2) for peripheral vowels. This serves as a baseline.
  • Class-Separability based on Formant Frequencies: Instead of just VSA, they propose a metric called "S" that measures the ratio of between-class (vowels) to within-class distances in the formant frequency space. This aims to capture how distinct vowels are from each other.
  • Vowel Spectral Metric (MFCCs): They replace formant frequencies with short-term spectral features, specifically Mel-frequency cepstral coefficients (MFCCs), to represent vowels. They then apply the class-separability metric "S" to these MFCC features. This method avoids the need for manual formant tracking.
  • Unsupervised Spectral Metrics: To eliminate the need for phonetic labeling altogether, they explore two unsupervised approaches using MFCCs:
  • Minimum Spanning Tree (MST) based Entropy: This method uses the structure of a minimum spanning tree built from the spectral features to estimate entropy, a measure of signal variation.
  • Hidden Markov Model (HMM) State Separability: They use an HMM to automatically segment the speech signal into different "states" (loosely corresponding to phonemes or contexts) and then apply the class-separability metric "S" to the features within these automatically derived states.

Datasets

The study uses a corpus of sentence productions from:

  • Healthy controls (CS)
  • Speakers with Multiple Sclerosis (MS)
  • Speakers with Parkinson's disease (PD)

These speakers read 24 Harvard sentences each. The acoustic signal was sampled at 22 kHz and down-sampled to 8 kHz. Intelligibility was measured in two ways:

  • Scaled Intelligibility (SI): Listeners rated intelligibility on a Visual Analog Scale (0.0 to 1.0), transformed to 1-SI for analysis.
  • Transcription Intelligibility (TI): Percentage of words correctly transcribed.

The paper notes that the SI and TI distributions were closely aligned (r²=0.82) and that the intelligibility scores spanned a relatively restricted range for all speakers, which they attribute to the mild nature of the dysarthria in the MS and PD groups.

Key Insights and Ideas

  • Automated Formant Tracking is Viable: Automatically extracted formant frequencies can provide a good estimate of Vowel Space Area (VSA), achieving a high correlation (r²=0.86) with manually corrected tracks.
  • Separability is Key: Simply measuring the "area" of feature space (like VSA) isn't as predictive as measuring how separable different classes (like vowels) are within that space. The proposed class-separability metric "S" shows better promise.
  • Spectral Features Outperform Formants: Using short-term spectral features (MFCCs) to measure vowel separability yielded better intelligibility prediction than using formant frequencies alone. This suggests that spectral details beyond just formant peaks are important.
  • Unsupervised Methods Struggle (in this study): While unsupervised methods (MST entropy, HMM state separability) are appealing for avoiding manual labeling, they did not perform as well in predicting intelligibility with this specific dataset and feature set compared to supervised spectral methods. The MST entropy measure, in particular, showed surprisingly poor results.
  • Mild Dysarthria is a Factor: The limited range of intelligibility scores and the moderate predictive power of the metrics are hypothesized to be due to the mild nature of the dysarthria in the clinical groups, making it harder to differentiate them acoustically.

Key Results

  • VSA (auto-formants) vs. VSA (manual-formants): r²=0.86.
  • Formant-based Class-Separability (S) vs. Intelligibility:
    • S → SI: r²=0.19 (p<0.01)
    • S → TI: r²=0.19 (p<0.01)
  • Vowel Spectral Metric (MFCCs) Class-Separability (S) vs. Intelligibility:
    • S → SI: r²=0.22 (p<0.01)
    • S → TI: r²=0.29 (p<0.01)
  • Unsupervised MST Entropy vs. Intelligibility:
    • H → SI: r²<0.01 (p>0.6)
    • H → TI: r²<0.01 (p>0.6)
  • Unsupervised HMM State Separability vs. Intelligibility:
    • S → SI: r² between 0.13 and 0.18 (for Q=20, p<0.01)
    • S → TI: r² between 0.10 and 0.13 (for Q=20, p=0.03)

The vowel spectral metric (MFCCs with class-separability) showed the strongest predictive power for intelligibility among the tested methods.

Limitations and Future Work

  • Mild Dysarthria: The study acknowledges that the mild nature of the dysarthria in the MS and PD groups limited the range of intelligibility scores and may have masked stronger relationships between acoustic features and intelligibility.
  • Unsupervised Method Performance: The MST entropy measure performed poorly, and the HMM state separability, while showing some promise, was not as strong as the supervised spectral methods. Future work could explore different unsupervised clustering or feature extraction techniques.
  • Feature Space Dimensionality: The paper mentions using 13th-order MFCCs and PCA for visualization. Further exploration of optimal feature dimensionality and combinations could be beneficial.
  • Speaking Styles: The authors plan to use additional speaking styles from the corpus to further investigate the metrics.
  • Clinical Populations: Expanding the study to include individuals with more severe dysarthria could provide clearer insights into the effectiveness of these metrics.
  • Clinical Adoption: The ultimate goal is to develop fully automated metrics that can be readily adopted in clinical practice for objective assessment of speech intelligibility.

Summary updated

BibTeX
@inproceedings{kain2017comparison,
  author = {Alexander Kain and Max Del Giudice and Kris Tjaden},
  title = {{A Comparison of Sentence-Level Speech Intelligibility Metrics}},
  booktitle = {Interspeech},
  year = {2017},
}
Figure