TL;DR
Schrödinger Bridge Consistency Trajectory Models (SBCTM) is a new approach to speech enhancement that combines the structural advantages of Schrödinger Bridges with the speed of Consistency Trajectory Models. It achieves high-quality, one-step speech denoising that is 16x faster than previous Schrödinger Bridge methods while outperforming them in perceptual quality.
What Problem Does This Paper Address?
Speech enhancement (SE) aims to remove background noise from recordings. Recently, diffusion models have become the gold standard for this, but they face two major hurdles:
- Prior Mismatch: Standard diffusion models assume the "noisy" end of the process is pure Gaussian white noise. In reality, noisy speech is a mix of clean speech and structured environmental noise. This mismatch leads to inefficient "bridges" between clean and noisy data.
- Slow Inference: Diffusion models typically require dozens or hundreds of sequential steps (Number of Function Evaluations, or NFE) to generate high-quality audio, making them too slow for real-time use.
While Schrödinger Bridges (SB) solve the first problem by directly modeling the path between clean and noisy speech distributions, they remain slow. Consistency Trajectory Models (CTMs) were designed to speed up image generation, but hadn't been successfully adapted to solve both the speed and mismatch issues in speech—until now.
Methods
The authors propose SBCTM, which integrates the CTM training framework into the Schrödinger Bridge.
Key technical components include:
- SB Framework: Instead of starting from random noise, the model learns a stochastic transport between the actual distribution of clean speech and noisy speech.
- Consistency Trajectory Distillation: The model is trained to predict any point along the "Probability Flow ODE" (the deterministic path of the diffusion process). This allows the model to "jump" from noisy speech to clean speech in a single step.
- Novel Auxiliary Loss: This is the "secret sauce." The authors added a differentiable PESQ (Perceptual Evaluation of Speech Quality) loss and a time-domain loss to the standard CTM training. This forces the model to prioritize human-audible quality rather than just minimizing mathematical pixel-wise (or spectrogram-wise) error.
- Architecture: They utilized the NCSN++ architecture, a common backbone for score-based generative models, modified with a time-condition embedding.
Datasets
The models were trained and evaluated on the VoiceBank+Demand (VB-DMD) dataset, a standard benchmark in the SE community.
- Training: ~9.5 hours of audio from 28 speakers.
- Testing: ~0.5 hours of official test data.
- Preprocessing: Audio was downsampled to 16 kHz and converted into complex STFT (Short-Time Fourier Transform) spectra.
Key Insights and Ideas
- The Power of One Step: SBCTM is designed so that its "one-step" performance is already excellent. Unlike standard consistency models, which can struggle to improve as you add more steps, SBCTM maintains a favorable trade-off between quality and speed across different step counts.
- Perceptual Tuning: The inclusion of the PESQ loss is critical. The authors found that while standard losses might improve signal-to-noise ratios (SI-SDR), they don't always make the speech sound "better" to humans. The auxiliary loss bridges this gap.
- Controllability: By adjusting the weight of the perceptual loss ($\lambda_p$), researchers can tune the model for different goals—prioritizing raw signal fidelity (for downstream tasks like speech recognition) or human-perceived clarity (for telephony).
Key Results
- Speedup: SBCTM achieved a Real-Time Factor (RTF) of 0.045, compared to 0.713 for the original SB-PESQ teacher model. This represents a 16x improvement in inference speed.
- Quality: At just 1 step (NFE=1), SBCTM achieved a PESQ score of 3.56, outperforming its own teacher model (3.50) and other competitive models like MP-SENet and CMGAN.
- Robustness: The model showed superior performance in DNSMOS (a non-intrusive neural-network-based quality metric), indicating that the enhanced speech sounds more natural to human-like evaluators.
Limitations and Future Work
- Metric Trade-offs: The authors noted that while PESQ scores were high, some other metrics like SI-SDR showed a slight decreasing trend as the number of inference steps increased. They attribute this to their "simple training strategies" and suggest more complex schedules could fix this.
- Model Size: With 66 million parameters, SBCTM is larger than some ultra-lightweight models (like DPT-FSNet at 0.88M), which might limit its use on very low-power edge devices.
- Expansion: The authors suggest that this framework could be extended beyond simple denoising to tasks like dereverberation (removing echoes) and declipping (fixing distorted audio).
Summary updated