Mel cepstral coefficient modification based on the glimpse proportion measure for improving the intelligibility of HMM-generated synthetic speech in noise

Cassia Valentini-Botinhao, Junichi Yamagishi, Simon KingInterspeech 2012

TL;DR

This paper introduces a novel method to enhance the intelligibility of synthetic speech generated by HMMs, particularly in noisy environments. By modifying Mel cepstral coefficients based on a "Glimpse Proportion" measure, they achieve significant improvements in speech clarity, even outperforming some Lombard speech adaptations.

What Problem Does This Paper Address?

When humans speak in noisy environments, they naturally adjust their speech to be more intelligible – a phenomenon known as Lombard speech. This paper aims to replicate this intelligibility enhancement in synthetic speech. Specifically, it tackles the challenge of improving the clarity of speech generated by Hidden Markov Model (HMM) based Text-to-Speech (TTS) systems when listeners are exposed to background noise. Existing methods either require extensive Lombard speech data for adaptation or rely on noise-independent modifications that may not be optimal. The authors seek a noise-dependent modification technique that can be applied during speech generation without needing prior knowledge of the specific noise.

Methods

The core of the proposed method lies in modifying the Mel cepstral coefficients (a common representation of speech spectral envelopes) using an objective intelligibility measure called the Glimpse Proportion (GP). The GP measure quantifies the proportion of time-frequency regions where speech is more prominent than noise, mimicking human listening strategies.

The authors adapt the GP measure to be a differentiable function, allowing for optimization. They then employ a Steepest Descent method to iteratively update the Mel cepstral coefficients to maximize the GP score for a given speech segment and noise.

Crucially, to control the amount of distortion and ensure audible speech quality, they introduce two key constraints:

  1. Limited Frequency Resolution: Modifications are restricted to the first few Mel cepstral coefficients. This is because these coefficients capture the broader spectral characteristics, while higher-order coefficients represent finer spectral details. By limiting modifications to the initial coefficients, they effectively limit the frequency resolution of the changes.
  2. Distortion Control: A stopping criterion is used, based on the mismatch between the auditory representations of modified and unmodified speech, to prevent excessive distortion.

Datasets

  • Training Data: Normal read-text speech data from a male speaker.
  • Lombard Speech Data: Speech data from the same male speaker recorded under noisy conditions (speech-modulated noise at 84 dBA over headphones).
  • Evaluation Data: The paper describes building eight different synthetic voices. Some were based on the normal speech data, while others were derived from the Lombard speech data. The evaluation involved listening tests with native English speakers.

Key Insights and Ideas

  • Objective Measure for Subjective Improvement: The paper demonstrates that an objective intelligibility measure (GP) can be effectively used to guide modifications that lead to significant subjective improvements in speech intelligibility.
  • The Power of Coarse Spectral Features: Modifying only the first two Mel cepstral coefficients (capturing broad spectral shape) can yield substantial intelligibility gains, comparable to or even exceeding modifications of all coefficients. This suggests that fine-grained spectral modifications might introduce detrimental distortions.
  • Lombard Speech vs. Modified Speech: While Lombard speech data naturally leads to more intelligible speech, the proposed Mel cepstral modification method can achieve similar or better intelligibility than Lombard speech spectral modifications, especially when only a few coefficients are altered.
  • Noise-Specific vs. Generalizable Modifications: The method allows for noise-dependent modifications without requiring explicit noise modeling for each specific noise type during feature extraction, making it more flexible.

Key Results

  • Speech-Shaped Noise: Modified voices (N-M10 and N-M2) achieved significantly higher Word Accuracy Rates (WAR) than the unmodified voice (N). N-M2 achieved 57.8% WAR compared to N's 40.9%. The Lombard voice (L) achieved 63.5% WAR.
  • Competing Talker Noise: The N-M2 voice performed better than the unmodified voice N (42.7% vs. 36.6% WAR) and was comparable to a Lombard-adapted voice (N-L, 43.6%). Lombard voices generally performed best in this condition as well.
  • Limited Modifications are Key: Modifying only the first few Mel cepstral coefficients (N-M2) was more effective than modifying all of them (N-M59) for intelligibility, especially in the competing talker scenario.

Limitations and Future Work

  • Lombard Speech Advantages: The authors acknowledge that Lombard speech offers intelligibility gains beyond spectral modifications, particularly from changes in duration, F0, and excitation parameters. Their method primarily focuses on spectral modifications.
  • Competing Talker Nuances: For competing talker noise, spectral changes seem less impactful than temporal adjustments. The authors suggest that for non-stationary noise, temporal energy re-allocation might be more beneficial.
  • Future Directions: The authors plan to explore energy reallocation across time and incorporate a loudness constraint rather than just an energy constraint for modifications. They also aim to conduct more extensive comparisons with other intelligibility enhancement methods.

Summary updated

BibTeX
@inproceedings{valentinibotinhao2012mel,
  author = {Cassia Valentini-Botinhao and Junichi Yamagishi and Simon King},
  title = {{Mel cepstral coefficient modification based on the glimpse proportion measure for improving the intelligibility of HMM-generated synthetic speech in noise}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure