TL;DR
This paper introduces a method for "morphing" between two different audio signals by treating their spectrograms as probability distributions and calculating their Wasserstein barycenter. By using Unbalanced Optimal Transport (UOT) and a custom cost matrix that limits how far sound energy can travel across time, the authors create hybrid sounds that preserve the rhythmic and textural character of the originals better than simple cross-fading.
What Problem Does This Paper Address?
When we want to transition between two sounds—say, a piano note and a guitar note—the standard approach is a "cross-fade" (Euclidean interpolation). Mathematically, this is just a weighted average. While simple, it results in a "ghostly" effect where both sounds play simultaneously at different volumes rather than truly merging into a new, hybrid sound.
Previous attempts to use Optimal Transport (OT)—a mathematical framework for moving "mass" (energy) from one distribution to another with minimum effort—have been limited. They often worked frame-by-frame, which ignores the temporal flow of music, or relied on specific models like Non-negative Matrix Factorization (NMF) that only work well for harmonic, pitched sounds. The authors aim to create a "global" interpolation method that works for any sound, from musical notes to noisy environmental textures like flowing water.
Methods
The authors propose a pipeline that moves beyond simple averaging:
- Spectrogram as Distribution: They take the Short-Time Fourier Transform (STFT) of the source and target sounds and treat the normalized magnitude spectrograms as 2D probability distributions over a time-frequency grid.
- Structured Cost Matrix: In standard OT, energy can move from any point in the source to any point in the target. To keep the sound "natural," the authors introduce a cost matrix $\bar{C}$ that penalizes moving energy too far across the time axis (controlled by a parameter $p$). If $p=0$, energy can only move across frequencies within the same time frame.
- Unbalanced Optimal Transport (UOT): Because the structured cost matrix might make it mathematically impossible to move all "mass" while satisfying strict constraints, they use UOT. This allows for some mass to be created or destroyed (penalized by Kullback-Leibler divergence), making the optimization more flexible and computationally efficient.
- Reconstruction: After finding the optimal "middle ground" distribution (the barycenter), they reassign the mass to a standard grid and use the Griffin-Lim algorithm to reconstruct the missing phase information, turning the spectrogram back into a listenable audio signal.
Datasets
The paper evaluates the method on two distinct scenarios using 1-second audio clips sampled at 16kHz:
- Synthetic Musical Notes: A C3-G3 sequence played on a piano (source) and a guitar (target), generated via Ableton Live.
- Environmental Sounds: A rhythmic cicada chirp (source) and the continuous texture of flowing water (target).
No large-scale benchmark datasets were used, as the focus was on the qualitative "creativity" of the interpolation.
Key Insights and Ideas
- Global vs. Local: By treating the spectrogram globally rather than frame-by-frame, the model can account for slight timing differences between sounds, leading to smoother transitions.
- The Power of $p$: The time-limiting parameter $p$ is the "secret sauce." Without it (standard OT), energy smears across the timeline, destroying the rhythm. With a small $p$, the model preserves the "envelope" or "beat" of the sound while morphing the tonal quality.
- UOT as a Stabilizer: Using Unbalanced OT isn't just a mathematical trick; it prevents the interpolation from becoming "muddy" by allowing the algorithm to "give up" on moving energy that would require too much distortion.
Key Results
- Hybrid Instruments: In the piano-to-guitar test, the UOT method ($p=0$) produced a convincing hybrid instrument that maintained the sharp attack and decay of the notes, whereas standard OT smeared the notes into a continuous, blurred drone.
- Texture Morphing: For the cicada and water example, the method successfully embedded the rhythmic "pulsing" of the cicada into the broad-spectrum noise of the water.
- Efficiency: By leveraging the sparsity of the structured cost matrix, the authors implemented a Majorization-Minimization (MM) algorithm that can process several seconds of audio on standard hardware, a significant improvement over naive OT solvers.
Limitations and Future Work
- Grid Reassignment: Currently, the "mass" ends up on an irregular grid and must be snapped back to a native grid. The authors suggest that working directly on irregular grids could improve fidelity.
- Phase Retrieval: The use of Griffin-Lim is a bottleneck for audio quality. The authors propose that future versions could use the phase information from the source and target signals to "inform" the phase of the interpolant.
- Real-time Constraints: While more efficient than previous OT methods, the process is not yet real-time, which would be necessary for live performance or interactive sound design tools.
Summary updated