Schrödinger Bridge Consistency Trajectory Models for Speech Enhancement

Shuichiro Nishigori, Koichi Saito, Naoki Murata, Masato Hirano, Shusuke Takahashi, Yuki Mitsufuji

Abstract: Speech enhancement (SE) utilizing diffusion models is a promising technology that improves speech quality in noisy speech data. Furthermore, the Schrödinger bridge (SB) has recently been used in diffusion-based SE to improve speech quality by resolving a mismatch between the endpoint of the forward process and the starting point of the reverse process. However, the SB still exhibits slow inference owing to the necessity of a large number of function evaluations (NFE) for inference to obtain high-quality results. While Consistency Models (CMs) address this issue by employing consistency training that uses distillation from pretrained models in the field of image generation, it does not improve generation quality when the number of steps increases. As a solution to this problem, Consistency Trajectory Models (CTMs) not only accelerate inference speed but also maintain a favorable trade-off between quality and speed. Furthermore, SoundCTM demonstrates the applicability of CTM techniques to the field of sound generation. In this paper, we present Schrödinger bridge Consistency Trajectory Models (SBCTM) by applying the CTM's technique to the Schrödinger bridge for SE. Additionally, we introduce a novel auxiliary loss, including a perceptual loss, into the original CTM's training framework. As a result, SBCTM achieves an approximately 16x improvement in the real-time factor (RTF) compared to the conventional Schrödinger bridge for SE. Furthermore, the favorable trade-off between quality and speed in SBCTM allows for time-efficient inference by limiting multi-step refinement to cases where 1-step inference is insufficient.

Introduction

Speech enhancement (SE) aims to recover clean speech from noisy recordings. While diffusion models have set new benchmarks for quality, they suffer from two primary limitations. First, the Prior Mismatch problem: standard diffusion assumes the terminal state is Gaussian noise, whereas noisy speech contains structured environmental noise. Second, Slow Inference: high-quality generation typically requires dozens or hundreds of sequential steps (Number of Function Evaluations, NFE), hindering real-time applications.

Schrödinger Bridges (SB) address the mismatch by modeling the direct path between clean and noisy speech distributions. However, they remain computationally expensive. Consistency Trajectory Models (CTMs) offer a solution for speed in image generation but require adaptation for the specific constraints of audio.

Methods: The SBCTM Framework

The proposed Schrödinger Bridge Consistency Trajectory Models (SBCTM) integrate CTM distillation into the SB framework. This allows the model to learn the "Probability Flow ODE"—the deterministic path connecting noisy and clean speech—and jump across this trajectory in as little as one step.

The SBCTM framework illustrates the trajectory from noisy speech ($x_T$) to clean speech ($x_0$), showing how the model can jump between time steps $t, u, s$ using the learned consistency function $G_\theta$.

A key technical innovation is the introduction of a novel auxiliary loss. By incorporating a differentiable PESQ (Perceptual Evaluation of Speech Quality) loss and a time-domain loss into the standard CTM training, the model is forced to prioritize human-audible clarity rather than just minimizing mathematical signal error. This "perceptual tuning" ensures that even one-step generations sound natural and clear.

Experimental Setup

The model was evaluated using the VoiceBank+Demand (VB-DMD) dataset. The training set comprised approximately 9.5 hours of audio from 28 speakers, while testing used the standard 0.5-hour test set. Audio was processed at 16 kHz using complex STFT spectra. The architecture utilized the NCSN++ backbone, modified for time-conditioned embeddings.

Results and Discussion

SBCTM demonstrates a significant leap in both speed and quality. It achieved a Real-Time Factor (RTF) of 0.045, a 16x improvement over the SB-PESQ teacher model (RTF 0.713).

Quantitative comparison of SBCTM against baseline models. SBCTM achieves state-of-the-art PESQ and DNSMOS scores even at low NFE counts.

As shown in the results table, SBCTM achieves a PESQ score of 3.56 at NFE=1, outperforming its teacher model and other competitive baselines. Furthermore, its high DNSMOS scores indicate superior naturalness. Unlike standard consistency models, SBCTM maintains high performance as NFE increases, allowing for flexible multi-step refinement when higher fidelity is required.

Conclusion

SBCTM successfully bridges the gap between the high-quality generation of Schrödinger Bridges and the inference speed of Consistency Models. By prioritizing perceptual quality through specialized loss functions, it enables high-fidelity, real-time speech enhancement. Future work may extend this framework to dereverberation and other audio restoration tasks.

Summary updated

Abstract

Speech enhancement (SE) utilizing diffusion models is a promising technology that improves speech quality in noisy speech data. Furthermore, the Schr\"odinger bridge (SB) has recently been used in diffusion-based SE to improve speech quality by resolving a mismatch between the endpoint of the forward process and the starting point of the reverse process. However, the SB still exhibits slow inference owing to the necessity of a large number of function evaluations (NFE) for inference to obtain high-quality results. While Consistency Models (CMs) address this issue by employing consistency training that uses distillation from pretrained models in the field of image generation, it does not improve generation quality when the number of steps increases. As a solution to this problem, Consistency Trajectory Models (CTMs) not only accelerate inference speed but also maintain a favorable trade-off between quality and speed. Furthermore, SoundCTM demonstrates the applicability of CTM techniques to the field of sound generation. In this paper, we present Schr\"odinger bridge Consistency Trajectory Models (SBCTM) by applying the CTM's technique to the Schr\"odinger bridge for SE. Additionally, we introduce a novel auxiliary loss, including a perceptual loss, into the original CTM's training framework. As a result, SBCTM achieves an approximately 16x improvement in the real-time factor (RTF) compared to the conventional Schr\"odinger bridge for SE. Furthermore, the favorable trade-off between quality and speed in SBCTM allows for time-efficient inference by limiting multi-step refinement to cases where 1-step inference is insufficient. Our code, pretrained models, and audio samples are available at https://github.com/sony/sbctm/.
BibTeX
@inproceedings{nishigori2025schrodinger,
  author = {Shuichiro Nishigori and Koichi Saito and Naoki Murata and Masato Hirano and Shusuke Takahashi and Yuki Mitsufuji},
  title = {{Schrödinger Bridge Consistency Trajectory Models for Speech Enhancement}},
  booktitle = {WASPAA},
  year = {2025},
  doi = {10.1109/waspaa66052.2025.11231011},
}
Figure