Intelligibility Prediction for Time-Modified Speech Signals Using Spectro-Temporal Modulation Features

TL;DR

This paper tackles the limitation of current speech intelligibility prediction (SIP) models that assume clean and degraded speech are perfectly time-aligned. They introduce a novel approach using spectro-temporal modulation (STM) features for more robust time alignment, which significantly improves the accuracy of intelligibility prediction, especially for speech that has been time-modified.

What Problem Does This Paper Address?

Predicting how intelligible speech is, especially in noisy or degraded conditions, is crucial for developing better communication technologies. Current methods, known as Reference-Based Speech Intelligibility Prediction Algorithms (RB-SIPAs), work by comparing a clean reference speech signal to its degraded version. However, a major limitation is their assumption that these two signals are perfectly aligned in time. This assumption breaks down when speech is processed in ways that alter its timing, such as in speech enhancement or when adapting speaking styles. This paper aims to overcome this limitation by developing a method to accurately align time-modified speech signals with their clean counterparts, thereby making RB-SIPAs more versatile and effective in real-world scenarios.

Methods

The core of this work lies in two main contributions:

1. Spectro-Temporal Modulation (STM) Features for Dynamic Time Warping (DTW): Traditional features like MFCCs struggle with time alignment in noisy conditions. The authors propose using STM features, which capture perceptually relevant aspects of speech, as input to DTW. STM analysis models the auditory system's processing of spectral and temporal modulations. By selecting specific STM channels that are robust to noise, they achieve more accurate time alignment paths compared to MFCC-based DTW. They also introduce a feature selection method to identify the most effective STM channels based on their contribution to alignment accuracy.
2. Incorporating Time Alignment into RB-SIPAs: Two methods are proposed to integrate the DTW-derived time alignment into existing RB-SIPAs:
3. Simple-SIP: This method uses the alignment path to align the time-frequency (TF) features extracted by the RB-SIPA, without altering the core SIP calculation.
4. Mean-SIP: This method goes a step further by averaging intermediate similarity scores across time frames that are repeated due to time scale modification, effectively creating a single score for such segments.

Datasets

The paper utilizes two datasets:

• Hurricane Challenge-I (HC-I): This dataset is used for evaluating the performance of the proposed time-aligned RB-SIPAs. It contains clean speech and degraded versions produced by additive noise at various Signal-to-Noise Ratios (SNRs), with some degraded signals having undergone time-scale modification.
• UrbanSound8k: This dataset is used in Experiment-I to evaluate the effectiveness of different STM channels for DTW alignment under various noise conditions and SNRs. It provides a diverse set of urban noises that are added to clean speech signals.

Key Insights and Ideas

• STM features are superior for time alignment: The research demonstrates that specific spectro-temporal modulation features are more effective than traditional MFCCs for accurately aligning time-modified speech signals, especially in the presence of noise.
• Perceptual relevance matters for alignment: The STM channels that are important for speech intelligibility also tend to be the most effective for time alignment, suggesting a strong link between perceptual cues and temporal synchronization.
• Time alignment significantly boosts intelligibility prediction: By incorporating accurate time alignment, the performance of RB-SIPAs can be substantially improved, leading to better correlation with human listening test scores.
• A few key STM channels are sufficient: The study found that selecting a small number of the most effective STM channels (around 2-3) yields significant improvements in alignment accuracy and subsequent SIP performance, avoiding the computational burden of using all available channels.

Key Results

• Improved Alignment Accuracy: STM features achieved significantly higher Percentage Agreement (PA) scores with MFCC-based alignment paths compared to using MFCCs alone, especially at lower SNRs. For example, at high SNR, STM features achieved around 83% agreement, while MFCCs alone were lower.
• Enhanced SIP Performance: When integrated into RB-SIPAs, the time-aligned STM features led to superior correlation with listening test scores on the Hurricane Challenge-I dataset. For instance, using STM-based alignment (specifically with 2-3 channels) resulted in higher Pearson, Spearman, and CCC scores compared to MFCC-based alignment and even the baseline without explicit time alignment.
• Mean-SIP Outperforms Simple-SIP: The Mean-SIP method, which averages over repeated frames, generally showed slightly better performance than Simple-SIP, indicating the benefit of explicitly handling time-scale modifications.

Limitations and Future Work

• Computational Cost of DTW: While FastDTW is used for efficiency, DTW itself can still be computationally intensive, especially for very long sequences.
• Threshold Selection: The choice of the threshold (τ) for path comparison in DTW alignment evaluation is somewhat arbitrary and might influence the results.
• Generalizability to Other Degradations: The study primarily focuses on noise and time-scale modifications. Further research could explore the effectiveness of STM features for alignment in the presence of other types of speech distortions.
• Exploring More RB-SIPAs: The proposed methods were tested on four existing RB-SIPAs. Future work could involve applying and evaluating these techniques with a wider range of SIP algorithms.
• Investigating the Role of Specific STM Channels: A deeper dive into why certain STM channels are more effective for alignment could lead to more targeted feature selection and potentially even more robust models.