Two-Dimensional Convolutional Recurrent Neural Networks for Speech Activity Detection

Anastasios Vafeiadis, Eleftherios Fanioudakis, Ilyas Potamitis, Konstantinos Votis, Dimitrios Giakoumis, Dimitrios Tzovaras, Liming Chen, Raouf HamzaouiInterspeech 2019

TL;DR

This paper proposes a novel approach to Speech Activity Detection (SAD) by treating it as a 2D image classification problem. By converting audio segments into spectrograms and feeding them into a Convolutional Recurrent Neural Network (CRNN), the authors achieve state-of-the-art results on a challenging dataset, outperforming existing methods.

What Problem Does This Paper Address?

Speech Activity Detection (SAD) is a crucial component in many audio processing applications, such as automatic speech recognition (ASR) and speaker diarization. The primary challenge in SAD is accurately distinguishing speech from non-speech segments, especially in real-world scenarios where audio is often degraded by various types of noise and overlapping speech. Traditional Voice Activity Detection (VAD) algorithms, while computationally efficient, often struggle in low Signal-to-Noise Ratio (SNR) environments and require extensive fine-tuning. While deep learning methods have shown promise, effectively handling the complex temporal and spectral characteristics of speech in noisy conditions remains an active research area.

Methods

The core innovation of this paper is framing SAD as a 2D multilabel image classification task. The authors convert 1-second audio segments into Short-time Fourier Transform (STFT) spectrograms, which are then used as input to a 2D Convolutional Recurrent Neural Network (CRNN).

Here's a breakdown of their approach:

  • Spectrogram Representation: Audio signals are transformed into grayscale spectrogram images using STFT. This captures both the frequency content and its evolution over time.
  • 2D CRNN Architecture: The proposed CRNN combines Convolutional Neural Network (CNN) layers for feature extraction and Recurrent Neural Network (RNN) layers (specifically Gated Recurrent Units - GRUs) for temporal modeling.
  • CNN Layers: Multiple convolutional layers with decreasing kernel sizes are used to extract hierarchical features from the spectrograms. Max-pooling operations are applied selectively in the frequency domain to preserve temporal information for the RNN. Batch normalization and ReLU activation functions are employed.
  • RNN Layers: The output of the CNN layers is reshaped and fed into bidirectional GRUs. These GRUs are adept at capturing long-term temporal dependencies in the sequence of features.
  • Post-processing: To mitigate the issue of "spikes" (short, misclassified segments) in the raw predictions, the authors apply two post-processing steps:
  • Moving Average Filter: A convolutional filter is applied to smooth the output, effectively averaging predictions over short time windows.
  • Minimum Duration Constraint: Segments predicted as speech that are shorter than a certain duration (150 ms) are relabeled as non-speech if their surrounding segments are also non-speech, leveraging the fact that human speech segments typically have a minimum duration.

Datasets

The experiments were conducted on the development and evaluation datasets from the 2019 Fearless Steps Challenge (Apollo 11). These datasets consist of approximately 30 minutes of audio per recording, sampled at 8000 Hz, and are characterized by naturalistic audio conditions.

Key Insights and Ideas

  • Framing SAD as 2D Image Classification: This is the central paradigm shift. By treating audio spectrograms as images, the authors leverage the power of CNNs, which are highly effective at image pattern recognition, for SAD.
  • Synergy of CNNs and RNNs: The combination of CNNs for local feature extraction (temporal and spectral) and RNNs for modeling long-term temporal dependencies proves highly effective for SAD.
  • Importance of Temporal Smoothing: The post-processing steps, particularly the moving average filter, are crucial for refining the raw predictions and significantly improving the Decision Cost Function (DCF) by removing spurious detections.
  • Robustness to Noise: The CRNN architecture, trained on spectrograms, demonstrates a remarkable ability to distinguish speech from non-speech even in challenging, noisy conditions, outperforming traditional VAD systems.

Key Results

The proposed 2D CRNN system achieved impressive results:

  • Development Set: A Decision Cost Function (DCF) of 2.89% was achieved on the development set of the 2019 Fearless Steps Challenge, representing a 66.4% improvement over the baseline system.
  • Evaluation Set: The system secured first place among 27 submissions on the evaluation dataset, with a DCF score of 3.318%.
  • Comparison: The 2D CRNN significantly outperformed other tested systems, including a 1D CRNN, an MFCC-based RNN, and the Google VAD system.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the DCF is calculated for each channel separately, and averaging these scores might be beneficial for a more comprehensive evaluation.
  • The current approach might not fully account for varying SNR levels across different channels.
  • Future Work:
    • Investigate per-channel DCF calculation and averaging.
  • Explore other deep learning architectures and techniques like pseudo-labeling and test-time augmentation.
  • Experiment with different audio segment lengths.
  • Embed the SAD approach into mobile devices and ASR systems.

Summary updated

BibTeX
@inproceedings{vafeiadis2019two,
  author = {Anastasios Vafeiadis and Eleftherios Fanioudakis and Ilyas Potamitis and Konstantinos Votis and Dimitrios Giakoumis and Dimitrios Tzovaras and Liming Chen and Raouf Hamzaoui},
  title = {{Two-Dimensional Convolutional Recurrent Neural Networks for Speech Activity Detection}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure