Whisper Activity Detection Using CNN-LSTM Based Attention Pooling Network Trained for a Speaker Identification Task

Abinay Reddy Naini, Malla Satyapriya, Prasanta Kumar GhoshInterspeech 2020

TL;DR

This paper introduces a novel Whisper Activity Detection (WAD) system that leverages a CNN-LSTM network, originally trained for speaker identification, to effectively identify whispered speech segments in noisy audio. The key innovation lies in using the attention weights from this speaker identification model to pinpoint whispered speech, demonstrating significant improvements over existing WAD methods, especially in challenging noisy and unseen environmental conditions.

What Problem Does This Paper Address?

Whisper Activity Detection (WAD) is the challenging task of identifying segments of whispered speech within audio recordings. Unlike standard Voice Activity Detection (VAD), WAD is more difficult due to the inherent characteristics of whispered speech: a lack of fundamental frequency (pitch) and a generally flatter spectral slope. These properties make it harder to distinguish whispered speech from background noise. While existing WAD methods have seen advancements, they often struggle in real-world scenarios with diverse noises and environmental conditions. A significant hurdle in developing better WAD systems is the scarcity of large, annotated datasets of whispered speech, which hinders the training of data-hungry deep learning models. This paper aims to address these limitations by proposing a WAD method that can be trained effectively even with limited annotated data and performs robustly across various noise conditions.

Methods

The core of the proposed WAD system is a CNN-LSTM based attention pooling network. This network is initially trained for speaker identification using whispered speech data. The rationale is that a model trained to distinguish speakers from whispered speech will implicitly learn to focus on the salient features of whispered speech itself.

Here's a breakdown of the method:

  • CNN Architecture: A convolutional neural network (CNN) is used as the initial feature extractor. It comprises three sequential CNN layers with ReLU activation and average pooling.
  • LSTM Attention Block: The output of the CNN is fed into an LSTM-based attention block. This block consists of two LSTM layers followed by an average pooling layer. The attention mechanism within the LSTM learns to assign weights to different frames of the whispered speech, effectively highlighting the most informative parts for speaker identification.
  • Speaker Identification Training: The entire CNN-LSTM network is trained for speaker identification using a categorical cross-entropy loss. The goal is to correctly classify the speaker from the whispered speech input.
  • WAD Decision: Crucially, the attention weights generated by the LSTM block during the speaker identification task are then repurposed for WAD. These weights, which indicate the importance of each frame for speaker identification, are passed through a post-processing pipeline to make the final WAD decision.
  • Post-processing: This pipeline involves:
  • Median Filter: To smooth out abrupt variations in the attention weights.
  • Binary Quantization: To convert the smoothed weights into binary values (0 or 1) based on learned thresholds.
  • Moving Average Window (MAW): To convert frame-level decisions into window-level decisions.
  • Summation: A logical OR operation to combine decisions and produce the final WAD output.

The key novelty is training a network for speaker identification and then leveraging its learned attention weights for WAD, thereby circumventing the need for extensive WAD-specific annotated data.

Datasets

The paper utilizes the following datasets:

  • WSPIRE: An in-house dataset containing whispered speech from 102 speakers, recorded across different dialect regions of India.
  • wTIMIT: A dataset comprising whispered speech from 28 North American and 20 Singaporean speakers.
  • CHAINS: A corpus with 36 speakers, including whispered speech recordings.

For noise conditions, the NOISEX-92 dataset was used, specifically employing eight different noise types (four for training and four for testing).

Key Insights and Ideas

  • Leveraging Speaker ID for WAD: The most significant insight is that a model trained for speaker identification can implicitly learn features crucial for WAD. The attention mechanism, by focusing on salient speech segments for speaker discrimination, effectively identifies regions likely to contain whispered speech.
  • Robustness to Unseen Conditions: The proposed method demonstrates superior performance in unseen noise conditions and environmental settings compared to baseline methods. This suggests that the attention-based approach is more generalized.
  • Reduced Annotation Dependency: A major advantage is the ability to fine-tune WAD parameters with only a small amount of annotated data, unlike baseline methods that might require extensive full training data annotation. This is critical given the scarcity of whispered speech datasets.
  • Attention Weights as a Proxy: The paper validates the idea that attention weights, typically used to highlight important input segments for a primary task (speaker ID), can serve as a reliable indicator for a secondary task (WAD).

Key Results

  • Experimental Setup 1 (Ex-1): The proposed APW method showed significant improvement over the baseline LTLEV, particularly in test cases 3 and 4 (unseen environments), where LTLEV performance dropped considerably. For high noise conditions (-10dB SNR), APW improved performance by approximately 9-21% over LTLEV.
  • Experimental Setup 2 (Ex-2): By augmenting the training data with noisy versions of whispered speech, APW further improved its performance, reaching up to ~96% CWAD for test case 1A. The improvement over Ex-1 was notable in test cases 3 and 4 (around 3.6-4.2%).
  • Noise Performance: APW showed the maximum average improvement (~7%) for white noise when noise augmentation was used during training. The method also demonstrated a significant advantage over LTLEV for MG noise at low SNR (-10dB), with a difference of up to ~22%.
  • Qualitative Results: Figure 5 visually demonstrates that the WAD decisions from the proposed method closely align with the ground truth, even at low SNRs, outperforming the baseline.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the proposed method is robust, its performance can be further improved with more annotated data.
  • The baseline LTLEV method's performance is highly dependent on optimal sub-band selection, which can be challenging for unseen conditions.
  • Future Work:
    • The authors propose exploring how the proposed method performs when trained for other speech tasks, such as gender classification.
  • Further investigation into more sophisticated post-processing methods could potentially enhance WAD performance.

Summary updated

BibTeX
@inproceedings{naini2020whisper,
  author = {Abinay Reddy Naini and Malla Satyapriya and Prasanta Kumar Ghosh},
  title = {{Whisper Activity Detection Using CNN-LSTM Based Attention Pooling Network Trained for a Speaker Identification Task}},
  booktitle = {Interspeech},
  year = {2020},
}
Figure