Real Time Speech Enhancement under Strict Latency Constraints

50 concepts45 papers1 proposal

When we communicate through a digital medium—whether using a smartphone, a VoIP application, or an assistive hearing aid—we expect two things simultaneously: pristine audio clarity and instant delivery. If an algorithm takes too long to suppress background noise, the resulting delay disrupts the natural cadence of human conversation, as noted by Kou Tanaka et al. in their work on assistive communication devices.

This tension creates a fundamental engineering challenge known as the latency-quality trade-off in speech enhancement. As highlighted by the organizers of the INTERSPEECH 2021 Deep Noise Suppression Challenge, modern deep learning models can achieve remarkable noise suppression, but deploying them in real-time interactive systems requires balancing this performance against strict computational and timing constraints.

To understand why this is difficult, we must look at where latency originates and how researchers are redesigning neural architectures to overcome it.


The Anatomy of Latency: Algorithmic vs. Computational

In real-time audio processing, latency is broadly split into two categories: algorithmic latency (the time required to accumulate enough audio data to process) and computational latency (the time the hardware takes to execute the model's mathematics).

The Frame-Size Dilemma

Traditional speech enhancement models operate in the frequency domain using the Short-Time Fourier Transform (STFT). To analyze frequencies accurately, the system must gather a "frame" of audio samples over time.

Algorithmic Latency Trade-off:
┌──────────────────────────────┐
│  Longer Frames (e.g., 32ms)  │ ──► Low Math Workload  ──► High Algorithmic Delay
└──────────────────────────────┘
┌──────────────────────────────┐
│  Shorter Frames (e.g., 2ms)  │ ──► High Math Workload ──► Low Algorithmic Delay
└──────────────────────────────┘

As Danilo de Oliveira et al. demonstrate in Efficient Transformer-based Speech Enhancement Using Long Frames and STFT Magnitudes, using longer frames (such as 32ms) significantly reduces the mathematical workload and memory footprint of the network because the model processes fewer total blocks per second. However, this introduces an unavoidable algorithmic delay: the system must wait 32ms just to fill the buffer before it can even begin processing.

Conversely, if you shrink the block size to minimize this delay, the computational overhead spikes dramatically. As Jan Franzen and Tim Fingscheidt observe in their work on in-car communication systems, reducing block sizes to achieve low delay forces adaptive filters to update much more frequently, creating a severe bottleneck for embedded hardware.

The Architectural Bottleneck

Even if the algorithmic delay is minimized, complex neural networks—such as multi-stream networks that map phase and harmonics (PHASEN) or multi-stage pipelines designed for video conferencing (Wei Rao et al. in the ConferencingSpeech Challenge)—often introduce high computational latency. These architectures require too many floating-point operations (FLOPs) to run within the millisecond-level windows required for live telecommunications (Zhuangqi Chen et al. in RaD-Net).


Bypassing the Window: Time-Domain and State-Space Models

To eliminate the algorithmic latency imposed by STFT windows, one prominent strategy is to abandon frequency-domain conversions entirely and process raw audio waveforms directly in the time domain.

Frequency Domain:  Raw Audio ──► STFT Window (Delay) ──► Neural Network ──► iSTFT ──► Clean Audio
Time Domain (SSM): Raw Audio ──► Continuous State Tracking (No Window Delay) ──► Clean Audio

A key breakthrough in this area leverages Deep State-Space Models (SSMs). In Optimized Real-time Speech Enhancement with Deep SSMs on Raw Audio, Yan Ru Pei et al. introduce aTENNuate, an architecture that processes raw audio sample-by-sample. Because SSMs maintain a compact, continuous internal memory state to track historical context, they do not need to wait for a large frame of audio to accumulate. This allows the system to maintain a tiny memory footprint and ultra-low latency, making it highly suitable for power-constrained edge devices like hearing aids.


Structural Efficiency: Subbands and Separable Convolutions

For systems that still rely on spectral transformations, researchers achieve real-time performance by optimizing the internal structure of the neural network.

  • Subband Splitting: Processing the entire frequency spectrum uniformly with heavy neural layers is computationally expensive. To solve this, Wenzhe Liu et al. developed Gesper, a framework that splits the spectrum into parallel subbands. By isolating specific frequency ranges, the model can apply complex spectral mapping where it matters most while keeping the overall latency within live communication limits.
  • Depthwise Separable Convolutions: Borrowing efficiency techniques from computer vision, Michal Romaniuk et al. introduced MASnet. By replacing standard convolutions with depthwise separable convolutions, they decoupled spatial filtering from channel mixing, drastically lowering the computational cost for mobile audio streaming.
  • Dual-Signal Stacking: The Dual-Signal Transformation LSTM Network (DTLN), proposed by Nils L. Westhausen and Bernd T. Meyer, stacks two distinct signal transformation methods. This architecture allows the network to successfully balance robust noise suppression with low computational complexity in a stacked, real-time loop (DTLN Network).

Accelerating Generative Models for Live Streams

Generative audio models (such as Diffusion models and Generative Adversarial Networks) produce incredibly natural-sounding speech, but they are traditionally slow. Diffusion models, for instance, require iteratively refining an audio signal over dozens of sequential steps, creating an inherent conflict with real-time delivery (Bunlong Lay et al.).

To bridge this gap, two primary mechanisms have emerged:

1. The Sliding Diffusion Buffer

Instead of running a full generation process from scratch for every chunk of audio, Bunlong Lay designed a Diffusion Buffer framework. This approach maps the iterative generation steps onto a continuous, sliding window of time. By reusing overlapping historical states, the model achieves sub-second latency (320–960 ms), making generative diffusion viable for live audio streaming on standard consumer laptops.

2. Single-Step Flow Matching

Another approach eliminates the iterative bottleneck entirely. In FlowAVSE, Chaeyoung Jung et al. utilize Conditional Flow Matching (CFM) to define a direct vector field from noisy to clean speech. Rather than stepping through a slow diffusion process, the model predicts the clean audio trajectory in a single inference step, yielding a real-time factor as low as 0.070.

A similar push for parallelization is seen in speech synthesis; Zalán Borsos et al. developed SoundStorm, which relies on a non-autoregressive, confidence-based parallel decoding scheme to generate high-fidelity audio orders of magnitude faster than traditional step-by-step autoregressive models.


The Ultra-Low Latency Frontier

For specialized applications like hearables and medical assistive listening devices, "low latency" is not enough; the system must operate in the ultra-low latency realm of under 5 milliseconds.

Latency Thresholds & Applications:
┌───────────────────────────┬────────────────────────────────────────┐
│ Latency                   │ Target Application                     │
├───────────────────────────┼────────────────────────────────────────┤
│ Sub-second (300-900ms)    │ Live Streaming, Generative Buffers     │
│ Low Latency (20-40ms)     │ VoIP, Video Conferencing, Mobile Apps  │
│ Ultra-Low Latency (<5ms)  │ Hearing Aids, Hearables                │
└───────────────────────────┴────────────────────────────────────────┘

In a comprehensive study of this frontier, Sebastian Braun and Haibin Wu systematically evaluated how models behave under these extreme constraints (Ultra-Low Latency Speech Enhancement). Their findings revealed a vital architectural intuition: when you force a model to operate with an ultra-short window, the signal quality naturally degrades due to the loss of temporal context.

Crucially, however, the authors demonstrated that increasing the model's parameter size is a highly effective way to recover this lost quality. While traditional signal processing techniques like asymmetric windows offer diminishing returns for deep networks, a larger, more expressive model can compensate for the lack of a temporal buffer by learning richer structural representations—proving that raw capacity can overcome the strict physical limitations of time.

Key Foundational Work

A new structural preprocessor for low-bit rate speech coding (2005) — contributed 1 concept elements to this direction.
Perceptual improvement of a two-stage algorithm for speech dereverberation (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_real_time_speech_enhancement_latency_906fcca5,
  title = {real-time speech enhancement latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-speech-enhancement-latency/rdr_906fcca57abf4d9eaf88c7080bafbc4c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_real_time_speech_enhancement_latency_906fcca5_v1,
  title = {real-time speech enhancement latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/real-time-speech-enhancement-latency/rdr_906fcca57abf4d9eaf88c7080bafbc4c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake