Speaker identification for whispered speech using modified temporal patterns and MFCCs

Xing Fan, John H. L. HansenInterspeech 2009 · DOI

Introduction: The Challenge of Whispered Speech

Speaker identification (ID) systems are typically optimized for "neutral" or normal speech. However, whispered speech presents a significant acoustic mismatch. Unlike normal speech, whispering lacks the periodic vibration of vocal folds (fundamental frequency, $F_0$), exhibits higher-frequency formants, and possesses a flatter spectral slope. This mismatch causes standard systems to fail, which is problematic for security and forensics applications where whispering is common for privacy or due to medical conditions like aphonia.

Comparison of spectrograms for neutral (a) and whispered (b) speech, highlighting the acoustic differences.

The core difficulty lies in the transition from high-energy voiced phonemes to low-energy, noisy signals. While spectral values change drastically, the researchers hypothesized that the temporal evolution of energy within a phoneme remains relatively stable across both speech modes.

Methodology: Modified Temporal Patterns (m-TRAPs)

To address the limitations of traditional Mel-Frequency Cepstral Coefficients (MFCCs), this paper proposes a hybrid system integrating "modified Temporal Patterns" (m-TRAPs). Unlike standard TRAPs that use fixed-length windows, m-TRAPs employ dynamic durations based on phoneme segmentation.

The proposed m-TRAP feature extraction architecture, showing phoneme segmentation and subband processing.

Key innovations in the m-TRAP approach include:

  1. Dynamic Phoneme-Based Windows: Features are extracted based on the actual duration of each speech sound rather than a fixed one-second window.
  2. Spectral Center Gravity (SCG): In addition to energy, the system tracks the SCG contour for each subband to capture vocal tract movement.
  3. Linear Filter Banks: 13 linear filter banks starting at 1000 Hz are used to better capture the high-frequency energy characteristic of whispers.
  4. Legendre Polynomials: 5th-order Legendre polynomials are used to smooth and represent the shapes of energy and SCG contours.

The system architecture utilizes 13 separate subband Gaussian Mixture Models (GMMs). During testing, a weighting strategy balances m-TRAP and MFCC scores based on the signal-to-noise ratio (SNR). If the SNR is low, the system relies more heavily on the m-TRAP scores; if the SNR is high, it leans back toward the MFCC scores.

Experimental Results

The system was evaluated using the UT-Whisper Corpus, featuring 10 female speakers. The hybrid approach was compared against a standard MFCC baseline across various conditions.

System Low SNR Whisper Accuracy Overall Whisper Accuracy
MFCC Baseline 6.0% 44.1%
m-TRAPs + MFCCs 57.5% 70.4%

The results demonstrate a massive 51.5% boost in accuracy for low-power whispers and a 26.3% absolute improvement overall. This suggests that temporal patterns—how energy rises and falls over a phoneme—are far more resilient to the acoustic distortions of whispering than short-term spectral features alone.

Conclusion and Future Directions

By focusing on the temporal "signature" of a speaker's articulation, this research provides a robust solution for whispered speaker identification. However, limitations remain, including the small sample size (10 female speakers) and the dependency on accurate phoneme segmentation in noisy environments. Future work will involve testing on larger, more diverse datasets and optimizing the computational complexity for real-time applications.

Summary updated

BibTeX
@inproceedings{fan2009speaker,
  author = {Xing Fan and John H. L. Hansen},
  title = {{Speaker identification for whispered speech using modified temporal patterns and MFCCs}},
  booktitle = {Interspeech},
  year = {2009},
  doi = {10.21437/interspeech.2009-270},
}
Figure