Longer features: they do a speech detector good

T. J. Tsai, Nelson MorganInterspeech 2012

TL;DR

This paper introduces spectrotemporal features, inspired by auditory cortex processing, for speech activity detection (SAD). These novel features significantly outperform traditional Mel-Frequency Cepstral Coefficients (MFCCs), achieving relative reductions in error rates of 40-50%, particularly when used with a simple Adaboost classifier. The findings suggest that features capturing longer temporal information are crucial for robust SAD.

What Problem Does This Paper Address?

Accurately detecting speech in noisy environments is a fundamental challenge in speech processing. Traditional methods often rely on features like MFCCs, which capture spectral information within short time frames. However, these features may not fully represent the complex ways the human auditory system processes sound, especially over longer durations. This paper aims to bridge this gap by exploring spectrotemporal features that mimic neural representations in the auditory cortex, hypothesizing that these "longer" features will lead to more robust speech detection, particularly in real-world noisy conditions.

Methods

The core of this work lies in the novel spectrotemporal features. These are computed by:

  1. Mel Spectrogram: First, the audio signal is transformed into a mel spectrogram, which represents the signal's energy across different frequency bands over time, scaled to the mel scale (mimicking human pitch perception).
  2. Gabor Filters: This mel spectrogram is then convolved with a set of 59 2D Gabor filters. These filters are designed to detect specific patterns of spectral and temporal modulations. Each filter can be thought of as a "spectrotemporal receptive field" (STRF), analogous to how neurons in the auditory cortex respond to specific sound patterns. The filters vary in their "size," capturing different combinations of spectral and temporal information.
  3. Critical Sampling: To reduce redundancy, only a subset of the filter outputs (449 features) are selected.
  4. Normalization: Finally, these features are normalized for each audio file.

For classification, three "back-ends" were used:

  • HMM-GMM: A standard Hidden Markov Model with Gaussian Mixture Models.
  • MLP: A Multi-Layer Perceptron that incorporates context from surrounding frames.
  • Adaboost: A boosting algorithm using simple "tree stump" weak classifiers, where each stump is a single feature compared to a threshold.

Datasets

The experiments were conducted on the Speech in Noisy Environments 2 (SPINE2) corpus. This dataset consists of recorded conversations between two participants in various real-world noisy environments (e.g., military settings, street noise, jet engine noise). Unlike some other datasets that artificially add noise to clean speech, SPINE2 captures more natural speech characteristics, including Lombard effects (speech changes in noisy environments) and acoustic reflections. Both training and evaluation sets contained approximately 7 hours of audio, split equally among 8 noise backgrounds.

Key Insights and Ideas

  • "Longer" Features Matter: The most striking insight is that spectrotemporal features, particularly those that capture information over longer time scales (0.5 to 1 second), are significantly more effective for speech activity detection than traditional MFCCs.
  • Adaboost Shines with Gabor Features: The Adaboost classifier, which is a simple, context-agnostic algorithm, performed exceptionally well with the Gabor spectrotemporal features. This suggests that the Gabor features themselves contain sufficient temporal information, making complex context-modeling less critical for this task.
  • MFCCs are "Skinny" and "Tall": MFCCs can be viewed as a special case of spectrotemporal features using very narrow (in the spectral dimension) and tall (in the temporal dimension) filters. This limits their ability to capture broader temporal patterns, which are shown to be crucial.
  • Biological Inspiration Pays Off: The spectrotemporal features, inspired by auditory cortex processing, demonstrate the power of drawing from neuroscience for signal processing tasks.

Key Results

  • Significant Performance Gain: Gabor spectrotemporal features consistently outperformed MFCCs across all tested back-ends.
  • Relative Error Reduction: Relative reductions in Equal Error Rate (EER) ranged from 40% to 50% when comparing Gabor features to MFCCs.
  • Adaboost Superiority: With Gabor features, Adaboost achieved the best performance, with an EER of 5.2%. This is notably better than the best MFCC performance (9.9% EER with HMM-GMM).
  • Longer Temporal Features Favored: Analysis of Adaboost's selected weak classifiers revealed a strong preference for features capturing low temporal modulation frequencies (below 3.9 Hz), corresponding to time spans of 0.5 to 1 second.

Limitations and Future Work

  • Adaboost Context Limitation: The paper acknowledges that Adaboost, by design, does not incorporate temporal context beyond what's present in the features themselves. While Gabor features mitigate this, it remains a limitation for other feature types.
  • Feature Selection: The Adaboost algorithm's ability to select the most relevant features is highlighted as a key advantage. Future work could explore how to explicitly design features that emphasize longer temporal scales.
  • Real-World Noise: The use of the SPINE2 corpus, with its real-world noise, is a strength. However, further exploration across a wider variety of acoustic conditions and noise types could be beneficial.
  • Computational Cost: While not explicitly detailed, the computation of Gabor filters might be more intensive than MFCCs. Further optimization or exploration of computationally lighter spectrotemporal features could be a direction.

Summary updated

BibTeX
@inproceedings{tsai2012longer,
  author = {T. J. Tsai and Nelson Morgan},
  title = {{Longer features: they do a speech detector good}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure