Gaussian Flow Bridges for Audio Domain Transfer with Unpaired Data

Eloi Moliner, Sebastian Braun, Hannes GamperarXiv 2024 · arXiv

TL;DR

This paper introduces Gaussian Flow Bridges (GFBs) for audio domain transfer, a method that modifies audio characteristics (like reverberation or distortion) without requiring paired "before and after" training data. By using a two-step deterministic flow process and a novel chunk-based minibatch Optimal Transport strategy, the authors achieve competitive results in speech restoration while preserving the original content's integrity.

What Problem Does This Paper Address?

Most state-of-the-art audio enhancement models rely on supervised learning, which requires massive datasets of paired samples (e.g., a clean recording and its exact reverberant counterpart). In the real world, obtaining these pairs is often expensive, difficult, or impossible.

While unsupervised methods like GANs or diffusion models exist, they face a major hurdle in audio: content consistency. When you ask a model to "remove reverb" without a ground-truth reference, it often hallucinates, changing the speaker's identity or making the speech unintelligible. This paper aims to bridge that gap, providing a way to perform domain transfer that is both unsupervised and content-preserving.

Methods

The core of the approach is the Gaussian Flow Bridge (GFB). Instead of trying to map one audio domain directly to another, the model uses a Gaussian distribution as an intermediate "bridge."

  1. Encoding: An unconditional vector field transforms the input audio ($x_0$) into a latent vector in a Gaussian noise space ($x_1$).
  2. Decoding: A conditional vector field then transforms that noise back into a modified audio sample ($\tilde{x}_0^c$), guided by a continuous control variable $c$ (such as a specific reverberation time $T_{60}$).
Fig. 1. (Top) Illustration of a GFB in one-dimensional space. (Mid- dle) A sequential display of spectrograms, showcasing the stages of audio signal transformation. (Bottom) Geometrical interpretation highlighting the mapping of data points through encoding and de- coding within a Gaussian space.

The "Straight Path" Innovation

A key challenge in flow-based models is that training trajectories can become curved and tangled, leading to artifacts. To fix this, the authors employ Flow Matching with linear interpolation. However, standard Optimal Transport (OT) couplings—used to keep these paths straight—don't scale well to high-dimensional audio waveforms.

The authors propose chunk-based minibatch OT. They partition long audio signals into small chunks (e.g., 32ms), solve the OT problem on these chunks to find the most efficient paths to noise, and then reshape them. This keeps the "trajectories" straight enough to preserve speech content without exploding computational costs.

Datasets

  • Training: Studio-quality speech from the VCTK corpus.
  • Reverberation: Speech was convolved with room impulse responses (RIRs) from multiple public datasets (including OpenAir and ACE), covering $T_{60}$ values from 0 to 1s.
  • Evaluation: The DAPS dataset (different from training) was used, convolved with real-world RIRs not seen during training to test generalization.
  • Distortion: Speech was artificially clipped at various Signal-to-Distortion Ratio (SDR) levels.

Key Insights and Ideas

  • Curvature is the Enemy: The authors demonstrate that "tangled" paths during training are a primary cause of identity shifts and speech artifacts. By using chunk-based OT, they significantly reduce trajectory curvature.
  • Linearity Preserves Content: Maintaining linear trajectories between data and noise is crucial for ensuring that the "content" (what is said) remains orthogonal to the "style" (the acoustic environment) being manipulated.
  • The Bridge Advantage: Using a Gaussian bridge allows for many-to-many mappings. You can take any reverberant signal and "bridge" it to a specific target $T_{60}$ or $C_{50}$ value just by changing the conditioning variable during the decoding step.
Fig. 2. Averaged trajectory curvature with respect to time τ when different coupling strategies are used. The shaded area represents the 25% and 75% percentiles.

Key Results

The GFB method was tested on two main tasks: dereverberation and declipping.

  • Dereverberation: The method performed competitively with supervised baselines like CRUSE and STORM. While it didn't strictly beat them, the fact that it achieved similar performance without paired data is a significant win.
  • Declipping: In speech declipping, the GFB significantly outperformed the popular SPADE baseline in terms of perceptual quality (NISQA scores).
  • Consistency: The use of chunk-based OT (C-OT) was shown to be critical. As shown in the objective evaluations, reducing the chunk size ($N_c$) generally improved speech consistency metrics like Word Error Rate (WER) and Speaker Recognition Cosine Similarity (SR-CS).
Fig. 4. Objective evaluation on speech dereverberation.

Limitations and Future Work

While promising, the method isn't perfect. The authors note occasional inconsistencies where the model might still slightly alter the speaker's identity or misinterpret a word. There is a clear trade-off: increasing the "guidance" to get more accurate acoustic target results often comes at the cost of introducing speech artifacts.

Future work could involve refining the chunk-based OT strategy or exploring more complex backbone architectures to further improve the robustness of the content preservation. The authors also suggest that this framework could be extended to other audio tasks, such as instrument timbre transfer or environmental sound synthesis.

Summary updated

Abstract

Audio domain transfer is the process of modifying audio signals to match characteristics of a different domain, while retaining the original content. This paper investigates the potential of Gaussian Flow Bridges, an emerging approach in generative modeling, for this problem. The presented framework addresses the transport problem across different distributions of audio signals through the implementation of a series of two deterministic probability flows. The proposed framework facilitates manipulation of the target distribution properties through a continuous control variable, which defines a certain aspect of the target domain. Notably, this approach does not rely on paired examples for training. To address identified challenges on maintaining the speech content consistent, we recommend a training strategy that incorporates chunk-based minibatch Optimal Transport couplings of data samples and noise. Comparing our unsupervised method with established baselines, we find competitive performance in tasks of reverberation and distortion manipulation. Despite encoutering limitations, the intriguing results obtained in this study underscore potential for further exploration.
BibTeX
@misc{moliner2024gaussian,
  author = {Eloi Moliner and Sebastian Braun and Hannes Gamper},
  title = {{Gaussian Flow Bridges for Audio Domain Transfer with Unpaired Data}},
  year = {2024},
  eprint = {2405.19497},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.19497},
}
Figure