Ultra-Low Latency Speech Enhancement - A Comprehensive Study

Haibin Wu, Sebastian BraunICASSP 2024 · DOI · OpenReview

TL;DR

This paper provides the first comprehensive, controlled study of ultra-low latency speech enhancement techniques. By unifying experimental setups and evaluating on large-scale, real-world data, the authors reveal that while some techniques like asymmetric windows offer little benefit for powerful models, they can be crucial for weaker ones. Increasing model size is shown to be a powerful way to recover performance lost due to reduced latency.

What Problem Does This Paper Address?

Modern hearables and assistive listening devices require speech enhancement (SE) systems to operate with extremely low latency, often under 5 milliseconds. Traditional SE models, like those based on the Short-Time Fourier Transform (STFT), typically have latencies of 20ms or more due to window lengths and hop sizes. While various low-latency techniques have been proposed, a lack of standardized evaluation across different methods makes it difficult to compare their effectiveness. Previous studies often used varying datasets, training procedures, and evaluation metrics, making fair comparisons impossible. Furthermore, many evaluations were conducted on small, simulated datasets, which may not accurately reflect real-world performance. This paper aims to bridge this gap by systematically evaluating several low-latency techniques under a unified framework and on realistic data.

Methods

The authors propose a unified framework to compare different low-latency SE techniques. Their base enhancement pipeline consists of an analysis transform, a DNN-based SE model (CRUSE, a Convolutional Recurrent U-Net), and a synthesis transform. They investigate the following low-latency strategies:

  • Asymmetric Windows: Using different window lengths for analysis and synthesis. This allows for a shorter synthesis window (reducing latency) while maintaining frequency resolution with a longer analysis window.
  • Learnable Analysis and Synthesis Transforms: Replacing fixed STFT basis functions with trainable convolutional layers. This allows the model to learn optimal transforms for the task.
  • Trainable Filterbank Equalizer (FBE): A time-domain processing technique using short Finite Impulse Response (FIR) filters. The latency is primarily determined by the filter length (hop size).
  • Future Frame Prediction: Predicting the next enhanced frame based on past observations. This can effectively reduce algorithmic latency, especially with overlapping frames.
  • Model Size Scaling: Investigating if increasing the model's capacity (e.g., number of convolutional filters) can compensate for performance degradation caused by reduced window sizes.
  • Mamba Architecture: Replacing the GRU bottleneck in their CRUSE model with the Mamba architecture to assess its performance in low-latency scenarios.

Datasets

  • Training Data: A large-scale dataset created by mixing 700 hours of high MOS-rated speech (LibriVox, AVspeech) with 247 hours of noise (Audioset, Freesound, internal noise sources, colored stationary noise).
  • Test Data: A combination of the 4th and 5th DNS Challenge blind test sets, plus an additional simulated test set generated similarly to the training data but with different corpora. For increased difficulty, the test set used only the lower signal-to-reverberation data portion.

Key Insights and Ideas

  • Asymmetric windows are not universally beneficial: For powerful models like CRUSE, asymmetric windows offer little to no advantage over symmetric ones, especially at higher latencies. However, they become significantly beneficial for weaker models (like their U-Net variant), suggesting they help compensate for limited model capacity.
  • Model size is a powerful latency compensation tool: Increasing model size can fully recover performance lost due to reduced window sizes. A large 3ms latency model performed comparably to the original 20ms latency model.
  • Future frame prediction has limitations: While it can reduce latency, it's restricted to mapping-based approaches, which underperform filtering-based methods. Its benefits are questionable in real-world scenarios, potentially struggling to generalize beyond simulated training data.
  • Mamba's low-latency performance is a concern: While Mamba excels at standard latency, its performance significantly degrades as latency is reduced.
  • FBE's trade-offs: FBE offers lower complexity and latency independent of window size, but it underperforms other techniques even when considering these advantages.

Key Results

  • Reducing latency from 20ms to 10ms had no significant impact on performance for symmetric windows with the CRUSE model. Performance degraded noticeably at 5ms and below.
  • Learnable transforms outperformed non-trainable STFT transforms at higher latencies, with differences diminishing at lower latencies.
  • The large 3ms latency model (F6) achieved performance comparable to the original 20ms latency model (A1), demonstrating the effectiveness of model scaling.
  • Asymmetric window models showed clear advantages over symmetric ones for the weaker U-Net model, but not for the CRUSE model.
  • Future-frame prediction models (G1, G2) performed worse than filtering-based models (A4, B3) across most metrics on the DNS blind test.

Limitations and Future Work

  • Limitations: The paper notes that future-frame prediction is limited to mapping-based approaches, which inherently underperform filtering-based methods. The FBE approach, while offering low latency, also underperforms. The Mamba architecture, while promising, struggles with low-latency performance.
  • Future Work: The authors suggest that their findings can guide future research in developing real-world low-latency SE systems. The insights into model scaling and the conditional benefits of asymmetric windows are particularly valuable. Further investigation into optimizing Mamba for low-latency scenarios or exploring other novel architectures could be promising.

Summary updated

Abstract

Speech enhancement models should meet very low latency requirements typically smaller than 5 ms for hearing assistive devices. While various low-latency techniques have been proposed, comparing these methods in a controlled setup using DNNs remains blank. Previous papers have variations in task, training data, scripts, and evaluation settings, which make fair comparison impossible. Moreover, all methods are tested on small, simulated datasets, making it difficult to fairly assess their performance in real-world conditions, which could impact the reliability of scientific findings. To address these issues, we comprehensively investigate various low-latency techniques using consistent training on large-scale data and evaluate with more relevant metrics on real-world data. Specifically, we explore the effectiveness of asymmetric windows, learnable windows, adaptive time domain filterbanks, and the future-frame prediction technique. Additionally, we examine whether increasing the model size can compensate for the reduced window size, as well as the novel Mamba architecture in low-latency environments.
BibTeX
@inproceedings{wu2024ultra,
  author = {Haibin Wu and Sebastian Braun},
  title = {{Ultra-Low Latency Speech Enhancement - A Comprehensive Study}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp49660.2025.10889823},
}
Figure