Abstract: Speech enhancement (SE) utilizing diffusion models is a promising technology that improves speech quality in noisy speech data. Furthermore, the Schrödinger bridge (SB) has recently been used in diffusion-based SE to improve speech quality by resolving a mismatch between the endpoint of the forward process and the starting point of the reverse process. However, the SB still exhibits slow inference owing to the necessity of a large number of function evaluations (NFE) for inference to obtain high-quality results. While Consistency Models (CMs) address this issue by employing consistency training that uses distillation from pretrained models in the field of image generation, it does not improve generation quality when the number of steps increases. As a solution to this problem, Consistency Trajectory Models (CTMs) not only accelerate inference speed but also maintain a favorable trade-off between quality and speed. Furthermore, SoundCTM demonstrates the applicability of CTM techniques to the field of sound generation. In this paper, we present Schrödinger bridge Consistency Trajectory Models (SBCTM) by applying the CTM's technique to the Schrödinger bridge for SE. Additionally, we introduce a novel auxiliary loss, including a perceptual loss, into the original CTM's training framework. As a result, SBCTM achieves an approximately 16x improvement in the real-time factor (RTF) compared to the conventional Schrödinger bridge for SE. Furthermore, the favorable trade-off between quality and speed in SBCTM allows for time-efficient inference by limiting multi-step refinement to cases where 1-step inference is insufficient.
Introduction
Speech enhancement (SE) aims to recover clean speech from noisy recordings. While diffusion models have set new benchmarks for quality, they suffer from two primary limitations. First, the Prior Mismatch problem: standard diffusion assumes the terminal state is Gaussian noise, whereas noisy speech contains structured environmental noise. Second, Slow Inference: high-quality generation typically requires dozens or hundreds of sequential steps (Number of Function Evaluations, NFE), hindering real-time applications.
Schrödinger Bridges (SB) address the mismatch by modeling the direct path between clean and noisy speech distributions. However, they remain computationally expensive. Consistency Trajectory Models (CTMs) offer a solution for speed in image generation but require adaptation for the specific constraints of audio.
Methods: The SBCTM Framework
The proposed Schrödinger Bridge Consistency Trajectory Models (SBCTM) integrate CTM distillation into the SB framework. This allows the model to learn the "Probability Flow ODE"—the deterministic path connecting noisy and clean speech—and jump across this trajectory in as little as one step.
A key technical innovation is the introduction of a novel auxiliary loss. By incorporating a differentiable PESQ (Perceptual Evaluation of Speech Quality) loss and a time-domain loss into the standard CTM training, the model is forced to prioritize human-audible clarity rather than just minimizing mathematical signal error. This "perceptual tuning" ensures that even one-step generations sound natural and clear.
Experimental Setup
The model was evaluated using the VoiceBank+Demand (VB-DMD) dataset. The training set comprised approximately 9.5 hours of audio from 28 speakers, while testing used the standard 0.5-hour test set. Audio was processed at 16 kHz using complex STFT spectra. The architecture utilized the NCSN++ backbone, modified for time-conditioned embeddings.
Results and Discussion
SBCTM demonstrates a significant leap in both speed and quality. It achieved a Real-Time Factor (RTF) of 0.045, a 16x improvement over the SB-PESQ teacher model (RTF 0.713).
As shown in the results table, SBCTM achieves a PESQ score of 3.56 at NFE=1, outperforming its teacher model and other competitive baselines. Furthermore, its high DNSMOS scores indicate superior naturalness. Unlike standard consistency models, SBCTM maintains high performance as NFE increases, allowing for flexible multi-step refinement when higher fidelity is required.
Conclusion
SBCTM successfully bridges the gap between the high-quality generation of Schrödinger Bridges and the inference speed of Consistency Models. By prioritizing perceptual quality through specialized loss functions, it enables high-fidelity, real-time speech enhancement. Future work may extend this framework to dereverberation and other audio restoration tasks.
Summary updated