Whispered Speech Detection Using Glottal Flow-Based Features

Khomdet Phapatanaburi, Wongsathon Pathonsuwan, Longbiao Wang, Patikorn Anchuen, Talit Jumphoo, Prawit Buayai, Monthippa Uthansakul, Peerapong UthansakulSymmetry 2022 · DOI

TL;DR

This paper introduces novel "Glottal Flow-based" features (GF-MFCC and GF-RP) to improve the detection of whispered speech, a task that significantly challenges standard Automatic Speech Recognition (ASR) systems. By analyzing the glottal flow signal, these new features capture distinct characteristics of whispered speech, leading to state-of-the-art performance when combined with existing methods.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) and Automatic Speaker Verification (ASV) systems, while highly effective for normal speech, often see a dramatic drop in performance when encountering whispered speech. This is because whispered speech production differs significantly from normal speech, particularly in vocal fold vibration and airflow. This mismatch makes it difficult for current systems to accurately process or classify whispered speech. The paper aims to address this by developing better features specifically designed to capture the unique acoustic properties of whispered speech, thereby improving the robustness of ASR/ASV systems. Beyond ASR/ASV, accurately detecting whispered speech has implications for diagnosing laryngeal diseases, understanding functional voice disorders, and developing ambient intelligence for smart environments.

Methods

The core innovation of this paper lies in the introduction of two new feature extraction methods that leverage the Glottal Flow (GF) signal. The GF signal represents the airflow from the glottis, which is a fundamental component of speech production. The authors propose:

  • GF-based Mel-Frequency Cepstral Coefficient (GF-MFCC): This feature modifies the standard MFCC extraction process. Instead of using the raw speech signal as input for the Mel filterbank and subsequent steps, it uses the GF signal estimated via the Iterative Adaptive Inverse Filtering (IAIF) technique. This aims to capture magnitude information more effectively for whispered speech.
  • GF-based Relative Phase (GF-RP): Similarly, this feature adapts the Relative Phase (RP) feature extraction by using the GF signal instead of the raw speech signal. This focuses on the phase information, which is also hypothesized to be distinct in whispered speech.

The GF signal itself is estimated using the IAIF technique, which iteratively refines glottal components and vocal tract transfer functions. The paper also explores feature-level and score-level combinations of these new GF-based features with conventional features (MFCC and RP) to further enhance detection performance. For classification, a Gaussian Mixture Model (GMM) with an Expectation-Maximization (EM) algorithm is employed, using a 5-fold cross-validation strategy due to the dataset's size.

Datasets

The study utilizes the publicly available CHAINS corpus. This corpus contains 1332 utterances of both normal and whispered speech from 36 speakers (16 female, 20 male) with three different accents. The utterances are sampled at 44.1 kHz and were downsampled to 16 kHz for computational efficiency. No new benchmarks were introduced; the focus was on evaluating existing and proposed features on this established dataset.

Key Insights and Ideas

  • Glottal Flow as a Differentiator: The fundamental insight is that the glottal flow signal, which is directly related to vocal fold vibration, carries crucial information that distinguishes whispered speech from normal speech. Whispered speech, lacking vocal fold vibration, has a different glottal source characteristic that is better captured by GF-based features.
  • Magnitude and Phase Complementarity: The paper effectively demonstrates that both magnitude (GF-MFCC) and phase (GF-RP) information derived from the GF signal are valuable for whispered speech detection. This suggests that these two aspects of the GF signal capture complementary information.
  • Augmentation, Not Replacement: The GF-based features are presented as augmentations to existing features (MFCC, RP). Combining them (feature-level) or their scores (score-level) proves more effective than using either alone, highlighting the benefit of integrating GF-derived information with traditional acoustic features.
  • GMM Limitations with Joint Features: The authors note that while combining magnitude and phase features can be beneficial, directly feeding joint magnitude and phase information into a GMM classifier can sometimes lead to worse performance than using individual features. This suggests that the GMM might struggle to model such complex joint distributions effectively, reinforcing the utility of score-level combinations.

Key Results

The proposed GF-based features significantly outperform their conventional counterparts:

  • GF-MFCC achieved higher accuracy and lower Equal Error Rate (EER) than MFCC.
  • GF-RP also showed superior performance compared to RP.
  • Feature-level combinations like MFCC&GF-MFCC and RP&GF-RP demonstrated substantial improvements over individual features.
  • The best frame-level performance was achieved by the combined score of MFCC&GF-MFCC and RP&GF-RP (MFCC&GF-MFCC+RP&GF-RP), yielding an accuracy of 95.01% and an EER of 4.85%.
  • At the utterance-level, the same combined system (MFCC&GF-MFCC+RP&GF-RP) achieved a perfect 100% accuracy and 100% F1-score, outperforming several deep learning-based systems on the CHAINS corpus.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the proposed systems show promising utterance-level performance, challenges remain for frame-level accuracy. The study also notes that the TECC feature, which incorporates both amplitude and frequency information of the raw signal, performed better than some of the proposed features. However, it's important to note that TECC was evaluated using self-selection training/testing datasets, whereas the proposed methods used a more robust five-fold cross-validation. The authors also mention that GMMs might not perform as well as deep learning classifiers on larger datasets, but deep learning was avoided due to the limited training data in the CHAINS corpus.
  • Future Work: The authors suggest investigating the effect of their proposed extraction method using empirical mode decomposition. Additionally, they propose exploring more efficient whispered voice activity detection algorithms to further improve frame-level classification accuracy.

Summary updated

BibTeX
@article{phapatanaburi2022whispered,
  author = {Khomdet Phapatanaburi and Wongsathon Pathonsuwan and Longbiao Wang and Patikorn Anchuen and Talit Jumphoo and Prawit Buayai and Monthippa Uthansakul and Peerapong Uthansakul},
  title = {{Whispered Speech Detection Using Glottal Flow-Based Features}},
  journal = {Symmetry},
  year = {2022},
  doi = {10.3390/sym14040777},
}
Figure