Target matching based generative model for speech enhancement

Taihui Wang, Rilin Chen, Tong Lei, Andong Li, Jinzheng Zhao, Meng Yu, Dong YuarXiv 2025 · DOI · arXiv

Abstract

Generative models for speech enhancement, such as Diffusion and Flow Matching, often struggle with computational inefficiency and the generation of artifacts, or "hallucinations." This paper introduces the Target Matching (TM) framework, which replaces stochastic vector fields with a deterministic target objective. Coupled with a novel Logistic Mean/Bridge Variance schedule and a Dual-path Spectro-temporal Diffusion Backbone (DBA), the proposed method achieves state-of-the-art performance on benchmarks like WSJ0-CHiME3. Notably, the model reduces parameters by 84% and computational costs by 33% compared to standard diffusion backbones, while enabling high-quality inference in as few as 4 steps.

Introduction

Speech enhancement aims to extract clean speech from noisy environments. While discriminative models are efficient, generative approaches like score-based diffusion models have gained traction for their ability to produce high-fidelity, natural audio. However, these models face three significant challenges:

  1. Inefficiency: Score-based and flow-matching models often require many iterative steps (NFEs) during inference.
  2. Hallucinations: Stochastic components in training objectives can cause models to generate speech-like artifacts in silent or low-SNR regions.
  3. Backbone Mismatch: Standard backbones like NCSN++ were designed for images and struggle to disentangle long-range temporal dependencies and harmonic frequency structures in audio.

Methodology

The authors propose a three-pronged solution to address these hurdles:

Target Matching (TM) Framework

Instead of training a network to predict a complex, stochastic vector field, the TM framework reformulates the task to predict the deterministic target signal ($x_0$) directly. This eliminates randomness in the loss function, leading to more stable training and faster convergence during inference.

Dual-path Spectro-temporal Diffusion Backbone (DBA)

The DBA backbone is designed specifically for audio. It utilizes a dual-path architecture to decouple temporal and frequency modeling:

  • T-blocks: Model long-term frame correlations (temporal).
  • F-blocks: Model cross-band dependencies (frequency).
The proposed TM-DBA architecture featuring the dual-path Time-Frequency block and timestep encoder.

Optimized Schedules

The authors introduce a Logistic Mean Schedule and a Bridge Variance Schedule. Unlike linear schedules, the logistic approach provides a more favorable Signal-to-Noise Ratio (SNR) trajectory. This preserves critical speech components early in the diffusion process and incorporates lower SNR conditions only at the end, effectively suppressing hallucinations.

Spectrogram evolution over timesteps comparing Exponential, Linear, and Logistic schedules. The Logistic schedule demonstrates superior preservation of speech structure.

Key Results

The proposed TM-DBA-M model was evaluated on the VoiceBank-DEMAND (VB-DMD) and WSJ0-CHiME3 benchmarks, demonstrating significant improvements:

  • Performance: On WSJ0-CHiME3, it achieved a PESQ of 2.80 and SI-SDR of 16.82, outperforming standard flow-matching (FM) and score-based models (SGMSE+).
  • Efficiency: The model is remarkably compact. Compared to the standard NCSN++ backbone, the DBA-M variant reduces parameters from 66M to 10.44M (84.2% reduction) and reduces FLOPs from 133G to 88.69G (33.3% reduction).
  • Inference Speed: While traditional generative models often need 10-20 steps to converge, the Target Matching approach achieves stable, high-quality output in just 4 steps.

Conclusion

The Target Matching framework effectively bridges the gap between the naturalness of generative models and the efficiency of discriminative ones. By treating audio as a distinct spectro-temporal signal and employing deterministic targets, the authors provide a robust solution for real-world speech enhancement. Future work will involve refining composite loss functions to perfectly balance perceptual quality with signal alignment.

Summary updated

BibTeX
@misc{wang2025target,
  author = {Taihui Wang and Rilin Chen and Tong Lei and Andong Li and Jinzheng Zhao and Meng Yu and Dong Yu},
  title = {{Target matching based generative model for speech enhancement}},
  year = {2025},
  eprint = {2509.07521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.07521},
}
Figure