Acoustic Analysis of Whispered Speech for Phoneme and Speaker Dependency

Xing Fan, Keith W. Godin, John H. L. HansenInterspeech 2011 · DOI

Introduction

Whispered speech presents a significant challenge for modern Automatic Speech Recognition (ASR) and Speaker Identification (SID) systems. Unlike neutral speech, whispering occurs without vocal fold vibration, resulting in a lack of glottal excitation. This physiological change leads to dramatic acoustic transformations: formants shift, bandwidths expand, and the spectral slope alters significantly. In this study, word accuracy dropped from 83.44% to 22.62% when transitioning from neutral to whispered speech. This paper investigates whether these transformations are consistent across speakers and phonemes or if they are highly individualized.

Methodology

The researchers modeled the transition from neutral ($ne$) to whispered ($wh$) speech as a linear time-invariant filtering process: $$wh(t) = ne(t) * h(t) + n(t)$$

To analyze the "whisper filter" ($h$), the study employed:

  • Vector Taylor Series (VTS): To map the distribution of Mel-Frequency Cepstral Coefficients (MFCCs) from neutral to whispered domains.
  • Expectation-Maximization (EM): To estimate hidden filter parameters for specific speaker-phoneme pairs.
  • Fisher’s Discriminant Power: To quantify whether transformations were more effective at distinguishing phonemes or speakers.

The primary dataset included 10 male native speakers of American English, each recording 10 sentences from the TIMIT corpus in both neutral and whispered modes.

Spectral Analysis and Speaker Dependency

The "whisper effect" varies across the frequency spectrum. Analysis was conducted across three subbands: S1 (0–2.7kHz), S2 (2.7–4kHz), and S3 (4–8kHz).

Comparison of log energy spectra for neutral and whispered speech across phonemes /ih/, /n/, and /z/ for ten speakers.

In lower frequencies (S1), the acoustic transformation is primarily driven by the phoneme. However, in higher frequencies (S3), individual speaker characteristics dominate. Fisher’s power for speaker discrimination ($F_s$) rose from 1.94 in S1 to 6.65 in S3, indicating that high-frequency whisper components are highly personalized. As shown in the spectral plots, while neutral speech remains relatively consistent, whispered speech exhibits significant inter-speaker variance, particularly in the upper frequency ranges.

Articulatory Correlations

A key finding is the relationship between physical articulation and spectral stability. Vowels with a high tongue position (e.g., /iy/, /uw/) exhibit lower inter-speaker variability in their whisper transformation compared to low vowels (e.g., /aa/).

Correlation between normalized tongue height and spectral variability across different vowel phonemes.

Furthermore, consonant types react differently to whispering. While voiced fricatives like /z/ remain relatively similar to their neutral counterparts, nasals (like /n/) and liquids experience significant energy compression in the 0–2kHz range. This suggests that the articulatory configuration of the vocal tract plays a major role in how whispering modifies the acoustic signal.

Conclusion

The high degree of speaker dependency in high-frequency spectral shifts suggests that universal whisper compensation models are insufficient. Future robust ASR systems must incorporate speaker adaptation methods to effectively bridge the performance gap between neutral and whispered speech. The study highlights that while some low-frequency shifts can be modeled generally, the high-frequency "fingerprint" of a whisper is unique to the individual.

Summary updated

BibTeX
@inproceedings{fan2011acoustic,
  author = {Xing Fan and Keith W. Godin and John H. L. Hansen},
  title = {{Acoustic Analysis of Whispered Speech for Phoneme and Speaker Dependency}},
  booktitle = {Interspeech},
  year = {2011},
  doi = {10.21437/interspeech.2011-77},
}
Figure