Schrödinger Bridge Consistency Trajectory Models for Speech Enhancement

Shuichiro Nishigori, Koichi Saito, Naoki Murata, Masato Hirano, Shusuke Takahashi, Yuki Mitsufuji

TL;DR

Schrödinger Bridge Consistency Trajectory Models (SBCTM) is a new approach to speech enhancement that combines the structural advantages of Schrödinger Bridges with the speed of Consistency Trajectory Models. It achieves high-quality, one-step speech denoising that is 16x faster than previous Schrödinger Bridge methods while outperforming them in perceptual quality.

What Problem Does This Paper Address?

Speech enhancement (SE) aims to remove background noise from recordings. Recently, diffusion models have become the gold standard for this, but they face two major hurdles:

  1. Prior Mismatch: Standard diffusion models assume the "noisy" end of the process is pure Gaussian white noise. In reality, noisy speech is a mix of clean speech and structured environmental noise. This mismatch leads to inefficient "bridges" between clean and noisy data.
  2. Slow Inference: Diffusion models typically require dozens or hundreds of sequential steps (Number of Function Evaluations, or NFE) to generate high-quality audio, making them too slow for real-time use.

While Schrödinger Bridges (SB) solve the first problem by directly modeling the path between clean and noisy speech distributions, they remain slow. Consistency Trajectory Models (CTMs) were designed to speed up image generation, but hadn't been successfully adapted to solve both the speed and mismatch issues in speech—until now.

Methods

The authors propose SBCTM, which integrates the CTM training framework into the Schrödinger Bridge.

Key technical components include:

  • SB Framework: Instead of starting from random noise, the model learns a stochastic transport between the actual distribution of clean speech and noisy speech.
  • Consistency Trajectory Distillation: The model is trained to predict any point along the "Probability Flow ODE" (the deterministic path of the diffusion process). This allows the model to "jump" from noisy speech to clean speech in a single step.
  • Novel Auxiliary Loss: This is the "secret sauce." The authors added a differentiable PESQ (Perceptual Evaluation of Speech Quality) loss and a time-domain loss to the standard CTM training. This forces the model to prioritize human-audible quality rather than just minimizing mathematical pixel-wise (or spectrogram-wise) error.
  • Architecture: They utilized the NCSN++ architecture, a common backbone for score-based generative models, modified with a time-condition embedding.

Datasets

The models were trained and evaluated on the VoiceBank+Demand (VB-DMD) dataset, a standard benchmark in the SE community.

  • Training: ~9.5 hours of audio from 28 speakers.
  • Testing: ~0.5 hours of official test data.
  • Preprocessing: Audio was downsampled to 16 kHz and converted into complex STFT (Short-Time Fourier Transform) spectra.

Key Insights and Ideas

  • The Power of One Step: SBCTM is designed so that its "one-step" performance is already excellent. Unlike standard consistency models, which can struggle to improve as you add more steps, SBCTM maintains a favorable trade-off between quality and speed across different step counts.
  • Perceptual Tuning: The inclusion of the PESQ loss is critical. The authors found that while standard losses might improve signal-to-noise ratios (SI-SDR), they don't always make the speech sound "better" to humans. The auxiliary loss bridges this gap.
  • Controllability: By adjusting the weight of the perceptual loss ($\lambda_p$), researchers can tune the model for different goals—prioritizing raw signal fidelity (for downstream tasks like speech recognition) or human-perceived clarity (for telephony).

Key Results

  • Speedup: SBCTM achieved a Real-Time Factor (RTF) of 0.045, compared to 0.713 for the original SB-PESQ teacher model. This represents a 16x improvement in inference speed.
  • Quality: At just 1 step (NFE=1), SBCTM achieved a PESQ score of 3.56, outperforming its own teacher model (3.50) and other competitive models like MP-SENet and CMGAN.
  • Robustness: The model showed superior performance in DNSMOS (a non-intrusive neural-network-based quality metric), indicating that the enhanced speech sounds more natural to human-like evaluators.

Limitations and Future Work

  • Metric Trade-offs: The authors noted that while PESQ scores were high, some other metrics like SI-SDR showed a slight decreasing trend as the number of inference steps increased. They attribute this to their "simple training strategies" and suggest more complex schedules could fix this.
  • Model Size: With 66 million parameters, SBCTM is larger than some ultra-lightweight models (like DPT-FSNet at 0.88M), which might limit its use on very low-power edge devices.
  • Expansion: The authors suggest that this framework could be extended beyond simple denoising to tasks like dereverberation (removing echoes) and declipping (fixing distorted audio).

Summary updated

Abstract

Speech enhancement (SE) using diffusion models is a promising technology for improving speech quality in noisy speech data. Recently, the Schrödinger bridge (SB) has been utilized in diffusion-based SE to resolve the mismatch between the endpoint of the forward process and the starting point of the reverse process. However, the SB inference remains slow owing to the need for a large number of function evaluations (NFE) to achieve high-quality results. Although consistency models have been proposed to accelerate inference, by employing consistency training via distillation from pretrained models in the field of image generation, they struggle to improve generation quality as the number of steps increases. Consistency trajectory models (CTMs) have been introduced to overcome this limitation, achieving faster inference while preserving a favorable trade-off between quality and speed. In particular, SoundCTM applies CTM techniques to sound generation. The SB addresses the aforementioned mismatch and CTM improves inference speed while maintaining a favorable trade-off between quality and speed. However, no existing method simultaneously resolves both issues to the best of our knowledge. Hence, in this study, we propose Schrödinger bridge consistency trajectory models (SBCTM), which apply CTM techniques to the SB for SE. In addition, we introduce a novel auxiliary loss, incorporating perceptual loss, into the original CTM training framework. Consequently, SBCTM achieves an approximately 16× improvement in real-time factor compared with the conventional SB for SE. Furthermore, the favorable trade-off between quality and speed in SBCTM enables time-efficient inference by limiting multi-step refinement to cases in which one-step inference is insufficient. SBCTM enables more practical and efficient speech enhancement by providing fast inference and a flexible mechanism for further quality improvement. Our codes, pretrained models, and audio samples are available at https://github.com/sony/sbctm/.
BibTeX
@inproceedings{nishigori2025schrodinger,
  author = {Shuichiro Nishigori and Koichi Saito and Naoki Murata and Masato Hirano and Shusuke Takahashi and Yuki Mitsufuji},
  title = {{Schrödinger Bridge Consistency Trajectory Models for Speech Enhancement}},
  booktitle = {WASPAA},
  year = {2025},
  doi = {10.1109/waspaa66052.2025.11231011},
}

Loading related content...

Figure