Multi-Level Adaptive Speech Activity Detector for Speech in Naturalistic Environments

Bidisha Sharma, Rohan Kumar Das, Haizhou Li

Introduction

Speech Activity Detection (SAD) is the critical first step in many audio processing pipelines, responsible for distinguishing human speech from background noise. While this task is relatively straightforward in controlled environments, it becomes exceptionally difficult in "naturalistic" settings. This paper addresses the extreme challenges posed by the Fearless Steps Challenge, which utilizes digitized analog communications from NASA’s Apollo-11 mission. These recordings represent a "worst-case scenario" for SAD, featuring far-field microphones, heavy ambient noise, hardware-induced harmonic interference, and highly variable speech densities.

The Limitations of Energy-Based Detection

Traditional SAD systems often rely on energy-based thresholds, assuming that speech is consistently louder than background noise. However, in naturalistic environments like the Apollo missions, this assumption fails. The energy distributions of speech and non-speech segments overlap significantly, making volume an unreliable metric for separation.

Energy and duration distributions for speech and non-speech segments, illustrating the significant overlap in normalized energy.

As shown in the figure above, the frequency of occurrences for normalized energy (plot a) reveals that speech and non-speech cannot be easily distinguished by energy alone. Furthermore, the duration of segments (plot b) varies wildly, with many files containing long stretches of silence punctuated by brief, bursty communications.

Proposed Methodology

To overcome these challenges, the researchers developed a multi-level, adaptive SAD system that focuses on the physical signatures of speech production rather than just volume. The architecture moves through several stages to refine the detection process.

Block diagram of the proposed multi-level adaptive SAD system architecture.

1. Denoising and Level 1 Filtering

The raw audio is first cleaned using spectral subtraction. The system then applies Modulation Spectrum (MS) analysis. This feature captures the rhythmic patterns of human speech (typically in the 2-12 Hz range), which corresponds to the movement of human articulators. Because most background noise lacks this specific rhythmic "pulse," MS is highly robust to environmental hums and static.

2. Level 2 Filtering: Smoothed Hilbert Envelope

For regions where speech is suspected but not confirmed, the system analyzes the Smoothed Hilbert Envelope (SHE) of the Linear Prediction (LP) residual. This technique tracks the "excitation source"—the vibration of the vocal folds—providing a second physical signature of speech.

3. Adaptive Q-Factor Thresholding

The most novel contribution of this work is the Quality Factor (Q-factor). The system calculates this factor by comparing the energy of the top 20% of frames to the bottom 20%. This allows the algorithm to automatically categorize a file as Sparse, Balanced, or Dense in terms of speech content. The system then adaptively adjusts its SHE and duration thresholds based on this context, allowing it to "read the room" and maintain sensitivity even in muffled or occasional communications.

Experimental Results

The system was evaluated using the Fearless Steps Challenge 2019 corpus, which includes audio from iconic phases like the lunar landing and moonwalk. Performance was measured using a Detection Cost Function (DCF), which penalizes missed speech more heavily than false alarms.

  • Proposed System: 7.35% DCF
  • Challenge Baseline: 11.70% DCF

The multi-level approach significantly outperformed the baseline and maintained a better balance between false positives and false negatives than standard statistical models. This performance earned the team a 9th-place ranking out of 27 international submissions.

Conclusion

By combining features that describe both speech production (Modulation Spectrum) and the vocal source (Hilbert Envelopes) with an adaptive thresholding mechanism, the researchers demonstrated that robust SAD is possible even in the most degraded analog environments. Future work will focus on automating the threshold tuning process through machine learning and testing the system's generalizability to other naturalistic datasets, such as body-cam footage and public space surveillance.

Summary updated

Abstract

Speech activity detection (SAD) is a part of many speech processing applications. The traditional SAD approaches use signal energy as the evidence to identify the speech regions. However, such methods perform poorly under uncontrolled environments. In this work, we propose a novel SAD approach using a multi-level decision with signal knowledge in an adaptive manner. The multi-level evidence considered are modulation spectrum and smoothed Hilbert envelope of linear prediction (LP) residual. Modulation spectrum has compelling parallels to the dynamics of speech production and captures information only for the speech component. Contrarily, Hilbert envelope of LP residual captures excitation source aspect of speech. Under uncontrolled scenario, these evidence are found to be robust towards the signal distortions and thus expected to work well. In view of different levels of interference present in the signal, we propose to use a quality factor to control the speech/non-speech decision in an adaptive manner. We refer this method as multi-level adaptive SAD and evaluate on Fearless Steps corpus that is collected during Apollo-11 Mission in naturalistic environments. We achieve a detection cost function of 7.35% with the proposed multi-level adaptive SAD on the evaluation set of Fearless Steps 2019 challenge corpus.
BibTeX
@inproceedings{sharma2019multi,
  author = {Bidisha Sharma and Rohan Kumar Das and Haizhou Li},
  title = {{Multi-Level Adaptive Speech Activity Detector for Speech in Naturalistic Environments}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1928},
}
Figure