EML Online Speech Activity Detection for the Fearless Steps Challenge Phase-III

Omid Ghahabi, Volker Fischer

Abstract

Speech Activity Detection (SAD), locating speech segments within an audio recording, is a main part of most speech technology applications. Robust SAD is usually more difficult in noisy conditions with varying signal-to-noise ratios (SNR). The Fearless Steps challenge has recently provided such data from the NASA Apollo-11 mission for different speech processing tasks including SAD. Most audio recordings are degraded by different kinds and levels of noise varying within and between channels. This paper describes the EML online algorithm for the most recent phase of this challenge. The proposed algorithm can be trained both in a supervised and unsupervised manner and assigns speech and non-speech labels at runtime approximately every 0.1 sec. The experimental results show a competitive accuracy on both development and evaluation datasets with a real-time factor of about 0.002 using a single CPU machine.

Introduction

Speech Activity Detection (SAD)—the task of identifying exactly when someone is speaking in an audio stream—is the "front door" for almost all speech technology, from transcription to speaker identification. While SAD is relatively straightforward in controlled environments, it becomes significantly more challenging in "naturalistic" settings with high noise levels.

The Fearless Steps Challenge (Phase-III) utilizes NASA Apollo-11 mission control tapes as a benchmark. This dataset presents a "perfect storm" of audio degradation, including varying signal-to-noise ratios, transmission distortions, analog tape hiss, and non-stationary noise. Researchers from EML Speech Technology have developed a high-performance, real-time SAD system specifically designed for these grueling acoustic environments. Their system achieves a nearly 4x improvement over the challenge baseline while maintaining an incredibly low computational footprint, processing audio 500x faster than real-time on a single CPU.

Methodology

The proposed "online" algorithm labels speech every 0.1 seconds using an architecture built on three main pillars:

  1. Two-Stage Feature Transformation: Instead of using raw Mel-Frequency Cepstral Coefficients (MFCCs), the system transforms them in two steps. First, a supervised Linear Discriminative Analysis (LDA) uses temporal context (concatenating frames from $t-10$ to $t+10$) to maximize the difference between speech and noise. Second, an unsupervised Principal Component Analysis (PCA) is applied to capture broader variations.
  2. Hybrid SAD Embeddings: The system represents speech and non-speech using low-dimensional vectors derived from zero-order Baum-Welch statistics and Deep Neural Network (DNN)-based embeddings.
  3. Online Adaptation: The system maintains buffer windows of recently detected speech and non-speech to continuously update model vectors and decision thresholds at runtime. This allows the system to adapt to the specific noise characteristics of the current audio stream on the fly.
The system architecture illustrating the training and testing pipelines, including the multi-stage feature transformation and automatic labeling.

Experimental Setup

The study utilizes the Fearless Steps Challenge Phase-III dataset, which includes manually labeled recordings from the NASA Apollo-11 mission. The training and development sets consist of 125 and 30 recordings (~30 mins each), respectively. The evaluation set includes 68 recordings scored via the NIST platform. The system is specifically tuned for the Decision Cost Function (DCF), which penalizes missed speech three times more heavily than false alarms, reflecting the priority of preserving vital mission communications.

Key Insights

  • Temporal Context: By concatenating multiple frames before applying LDA and PCA, the model gains a "temporal view" that helps distinguish human speech from short bursts of static.
  • Statistical Efficiency: Extracting Baum-Welch statistics compresses the data significantly, allowing a very small DNN to be highly effective and computationally efficient.
  • Dynamic Adaptation: Adapting models and thresholds at runtime improved accuracy by 5–10%, demonstrating that static models are at a disadvantage in highly variable environments.

Results

The EML algorithm significantly outperformed both the challenge baseline and other established methods. As shown in the table below, the system achieved a DCF of 3.80% on the evaluation set, a substantial improvement over the 15.61% baseline.

Performance comparison showing the EML Online Algorithm's superior DCF scores on both Development (Dev) and Evaluation (Eval) datasets.

Beyond accuracy, the system is exceptionally fast, achieving a Real-Time Factor (RTF) of 0.002. This allows it to process 1,000 seconds of audio in just 2 seconds on a single 2.30 GHz CPU, making it ideal for large-scale pre-processing.

Conclusion and Future Work

The combination of zero-order statistics and DNN embeddings proved particularly effective for utterances with very low signal-to-noise ratios. While the current system is highly robust, future work will focus on better utilizing the massive 19,000-hour unlabeled Apollo-11 corpus to further refine the embeddings. Additionally, the researchers aim to address potential overfitting risks associated with the relatively small development set by exploring more diverse training data.

Summary updated

Abstract

Speech Activity Detection (SAD), locating speech segments within an audio recording, is a main part of most speech technology applications. Robust SAD is usually more difficult in noisy conditions with varying signal-to-noise ratios (SNR). The Fearless Steps challenge has recently provided such data from the NASA Apollo-11 mission for different speech processing tasks including SAD. Most audio recordings are degraded by different kinds and levels of noise varying within and between channels. This paper describes the EML online algorithm for the most recent phase of this challenge. The proposed algorithm can be trained both in a supervised and unsupervised manner and assigns speech and non-speech labels at runtime approximately every 0.1 sec. The experimental results show a competitive accuracy on both development and evaluation datasets with a real-time factor of about 0.002 using a single CPU machine.
BibTeX
@inproceedings{ghahabi2021eml,
  author = {Omid Ghahabi and Volker Fischer},
  title = {{EML Online Speech Activity Detection for the Fearless Steps Challenge Phase-III}},
  booktitle = {Interspeech},
  year = {2021},
}
Figure