Making conversational vowels more clear

Seyed Hamidreza Mohammadi, Alexander Kain, Jan P. H. van SantenInterspeech 2012

TL;DR

This paper introduces a novel speech transformation technique to make conversational speech clearer, aiming to improve intelligibility for individuals with hearing impairments or in noisy environments. By mapping conversational speech spectral features to those of "clear" speech, they demonstrate significant improvements in vowel recognition, particularly in challenging listening conditions.

What Problem Does This Paper Address?

Understanding speech in noisy or distracting environments is a significant challenge, especially for individuals with hearing loss. While hearing aids can amplify sound, they don't always address the underlying acoustic degradations that reduce speech intelligibility. Speakers naturally adopt a "clear" speech style when they perceive a listener might have difficulty understanding, characterized by more distinct vowel sounds, longer durations, and clearer articulation. This paper tackles the problem of automatically transforming standard "conversational" speech into this clearer, more intelligible form, bridging the gap between how we naturally speak and how we should speak for optimal comprehension.

Methods

The core of this work is a speech transformation method that maps the spectral features of conversational (CNV) speech to resemble clear (CLR) speech. They explore two domains for this mapping:

  • Formant Frequency Domain: This approach uses the first two formant frequencies (F1 and F2), which are fundamental to vowel acoustics. This domain is chosen for its low dimensionality, making it easier to visualize and understand the transformation process.
  • Line Spectral Frequency (LSF) Domain: This method uses higher-dimensional LSF parameters, offering a more comprehensive spectral representation for a more robust transformation.

For the mapping function, they employ a joint-density Gaussian mixture model (JDGMM). This is a machine learning technique that learns the statistical relationship between CNV and CLR speech features. The process involves:

  1. Feature Extraction: Identifying vowel centers and extracting spectral features (formants or LSFs).
  2. Training: Using a portion of the recorded data to train the JDGMM, learning how to transform CNV features to CLR features.
  3. Mapping: Applying the trained JDGMM to new CNV speech data to generate transformed spectral features.
  4. Synthesis: Reconstructing the speech signal from the transformed features (though the paper focuses on the spectral mapping itself).

They also investigate the impact of vowel duration, creating a combined "MAP-SD" condition that incorporates both spectral and duration modifications.

Datasets

The study utilizes a custom corpus recorded specifically for this research:

  • Corpus: 242 American English sentences spoken by a single male speaker.
  • Styles: Each sentence was recorded in both conversational (CNV) and clear (CLR) speech styles.
  • Vowel Focus: The analysis and listening tests specifically focus on CVC (consonant-vowel-consonant) words, with an emphasis on the eight monophthong vowels.
  • Training/Testing Split: The data was split into training and testing sets using k-fold cross-validation (k=10) for model training and evaluation.

Key Insights and Ideas

  • Vowel Space Expansion: Clear speech demonstrably expands the acoustic "vowel space," making individual vowels more distinct and less prone to overlap compared to conversational speech. This is a key acoustic correlate of intelligibility.
  • Mapping Complexity: The relationship between CNV and CLR vowel features is not uniform across all vowels. Some vowel transformations are more consistent than others, suggesting that a single, simple mapping might not be optimal for all cases. Regions of high inconsistency in the mapping highlight areas where vowel overlap is most problematic.
  • Spectral Mapping Effectiveness: The spectral transformation alone (MAP-S) shows promise, leading to improved intelligibility, especially in noisier conditions.
  • Duration Matters: Combining spectral mapping with duration modifications (MAP-SD) yields the most significant intelligibility gains, suggesting that both spectral clarity and temporal distinctiveness are crucial for enhanced comprehension.
  • Real-World Applicability: The spectral mapping (MAP-S) is highlighted as a practical approach for real-time applications like hearing aids, where manipulating speech duration might be technically challenging due to video synchronization issues.

Key Results

  • Vowel Space: Figures 1 and 3 visually confirm that CLR speech has a larger and more separated vowel space than CNV speech.
  • Mapping Performance (Spectral Domain): The JDGMM mapping in the LSF domain (Figure 4c) shows a good approximation of the reference CLR relationship, with average Log Spectral Distance (LSD) errors significantly lower than unmapped CNV speech (Table 2).
  • Intelligibility Testing (Table 3):
    • At a challenging -2 dB SNR, the MAP-SD condition (spectral + duration mapping) showed significantly better intelligibility than CNV and vocoded CNV (VCNV) speech.
  • The MAP-S condition (spectral mapping only) also showed improvement over CNV and VCNV, though not statistically significant at -2 dB SNR.
  • At a less noisy +3 dB SNR, both MAP-S and MAP-SD conditions showed significant improvements over CNV and VCNV.
  • Contribution of Features: The results suggest that spectral and duration mappings contribute independently to intelligibility, and their combined effect is additive.

Limitations and Future Work

  • Single Speaker: The study used only one male speaker for the corpus. Future work should investigate if the learned mappings generalize across different speakers and genders.
  • Vowel Overlap Regions: The paper acknowledges that certain vowel regions exhibit high inconsistency in the mapping. Future work could explore strategies to address these specific areas, potentially by incorporating contextual information or more advanced modeling techniques.
  • Real-time Implementation: While MAP-S is deemed practical for real-time applications, further research is needed to fully integrate and optimize these transformations for devices like hearing aids.
  • Parametric Duration Modeling: For applications where duration manipulation is feasible (e.g., offline video processing), the paper suggests developing parametric duration models to work in conjunction with spectral mapping for MAP-SD transformations.

Summary updated

BibTeX
@inproceedings{mohammadi2012making,
  author = {Seyed Hamidreza Mohammadi and Alexander Kain and Jan P. H. van Santen},
  title = {{Making conversational vowels more clear}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure