Neural Whispered Speech Detection with Imbalanced Learning

Takanori Ashihara, Yusuke Shinohara, Hiroshi Sato, Takafumi Moriya, Kiyoaki Matsui, Takaaki Fukutomi, Yoshikazu Yamaguchi, Yushi AonoInterspeech 2019

TL;DR

This paper tackles the challenge of accurately detecting whispered speech, a difficult task due to the scarcity of whispered speech data and the resulting class imbalance in training datasets. They propose a novel approach using a class-aware sampling method with Convolutional Neural Networks (CNNs) to effectively train models even with limited whispered speech examples, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Whispered speech is an increasingly important mode of human-computer interaction, especially in environments where loud speech is inappropriate or privacy is a concern (e.g., libraries, meetings). However, existing speech processing systems, like automatic speech recognition and speaker recognition, are primarily trained on normally phonated speech. This leads to a significant performance degradation when encountering whispered speech. While a whispered speech detector is crucial for enabling such systems to effectively handle whispers (e.g., Alexa's whispered response mode), training these detectors is hampered by a fundamental problem: the extreme scarcity of whispered speech data compared to normal speech. This class imbalance makes it difficult for machine learning models to learn effectively, often leading to models that are biased towards the majority class (normal speech) and perform poorly on the minority class (whispered speech).

Methods

The core of this paper's contribution lies in its approach to handling the imbalanced data problem for whispered speech detection. They leverage Deep Neural Networks (DNNs), specifically Convolutional Neural Networks (CNNs), which have shown promise in automatically learning relevant features.

The key innovation is the application of class-aware sampling during the training phase. This technique, adapted from existing oversampling methods, ensures that mini-batches fed to the neural network contain an equal number of whispered and non-whispered speech samples. This is achieved by sampling from the minority class more frequently, effectively creating a balanced training set without discarding valuable majority class data.

Furthermore, the paper focuses on utterance-level classification, meaning the model predicts whether an entire speech segment is whispered or not, rather than classifying individual frames. To achieve this, they employ a CNN architecture that incorporates a global max-pooling layer after the convolutional layers. This layer aggregates temporal features, allowing the network to learn utterance-level representations and optimize for utterance-level inference. They also compare their CNN approach with traditional frame-level models like Multilayer Perceptrons (MLPs) and Long Short-Term Memory (LSTM) networks, which require separate inference modules for utterance-level prediction.

Datasets

The experiments utilized a combination of existing and newly created datasets:

  • CHAracterizing INdividual Speakers (CHAINS) [23]: This corpus was used for its speaking styles, specifically "WHISPER reading" and "SOLO reading," to extract whispered speech. It was also used to create a "balanced class distribution" dataset for some experiments.
  • Corpus of Spontaneous Japanese (CSJ) [24]: This corpus, consisting of academic presentations and simulated public speech, was treated entirely as non-whispered speech data. It was combined with CHAINS to create an "imbalanced class distribution" dataset.
  • Newly Created Test Set: This dataset was specifically recorded to simulate practical environments. It includes speech captured by both condenser and smartphone microphones at varying distances (3 cm, 15 cm, 50 cm) from the speaker, encompassing both neutral and whispered speech modes. This dataset was balanced for testing purposes.

Key Insights and Ideas

  • Class Imbalance is a Major Hurdle: The paper strongly demonstrates that training whispered speech detectors on severely imbalanced datasets leads to degraded performance, particularly in accurately identifying whispered speech.
  • Class-Aware Sampling is Effective: The proposed class-aware sampling method significantly enhances the performance of utterance-level classifiers when dealing with imbalanced data, especially in challenging conditions like far-field recordings.
  • CNNs Excel at Utterance-Level Detection: CNNs, with their ability to learn hierarchical features and the incorporation of global max-pooling, are well-suited for direct utterance-level whispered speech detection, outperforming traditional frame-level models.
  • Robustness to Microphone Variability: The CNN architecture shows a degree of robustness to variations in microphone types, suggesting that the max-pooling layer effectively mitigates spectral variations.
  • Far-Field Performance is Challenging: While the proposed method improves performance, far-field conditions (50 cm) remain the most challenging for whispered speech detection due to acoustic differences between training and testing environments.

Key Results

The experimental results highlight the effectiveness of the proposed approach:

  • Superior Performance with Imbalanced Learning: The utterance-level CNN trained with class-aware sampling (CNN (imbalance)) consistently outperformed baseline models (MLP, LSTM) and the CNN trained on balanced data across various conditions.
  • Near-Perfect Accuracy in Ideal Conditions: In close and neutral distance conditions (3 cm and 15 cm) with condenser microphones, the CNN (imbalance) achieved ROC-AUC scores close to 1.0 (e.g., 99.96% at 15 cm).
  • Significant Improvement in Far-Field: Even in the challenging far-field (50 cm) condition with smartphone microphones, the CNN (imbalance) achieved an ROC-AUC score of 91.68%, a notable improvement over the CNN with balanced data (73.09%).
  • Smartphone Microphone Performance Degradation: Overall, performance was lower with smartphone microphones compared to condenser microphones, likely due to the training data being primarily captured by condenser microphones.

Limitations and Future Work

  • Limitations:
    • The performance degradation in far-field conditions (50 cm) indicates that acoustic variability, especially at greater distances, remains a challenge.
  • The performance difference between condenser and smartphone microphones suggests that training data diversity is crucial for real-world robustness.
  • Future Work:
    • Explore other training methods for imbalanced data, such as cost-sensitive training and one-class classification.
  • Investigate data augmentation techniques like speed perturbation and pseudo-whisper generation to further compensate for acoustic variability and improve robustness.

Summary updated

BibTeX
@inproceedings{ashihara2019neural,
  author = {Takanori Ashihara and Yusuke Shinohara and Hiroshi Sato and Takafumi Moriya and Kiyoaki Matsui and Takaaki Fukutomi and Yoshikazu Yamaguchi and Yushi Aono},
  title = {{Neural Whispered Speech Detection with Imbalanced Learning}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure