Audio signal interpolation using optimal transportation of spectrograms

David Valdivia, Marien Renaud, Elsa Cazelles, Cédric Févotte

Abstract

We present a novel approach for generating an artificial audio signal that interpolates between given source and target sounds. Our approach relies on the computation of Wasserstein barycenters of the source and target spectrograms, followed by phase reconstruction and inversion. In contrast with previous works, our new method considers the spectrograms globally and does not operate on a temporal frame-to-frame basis. Another contribution is to endow the transportation cost matrix with a specific structure that prohibits remote displacements of energy along the time axis, and for which optimal transport is made possible by leveraging the unbalanced transport framework. The proposed cost matrix makes sense from the audio perspective and also allows to reduce the computation load. Results with synthetic musical notes and real environmental sounds illustrate the potential of our novel approach.

Introduction

Audio morphing, the process of creating a seamless transition between two distinct sounds, is a challenging task in digital signal processing. Traditional methods, such as Euclidean interpolation (cross-fading), typically result in a simple superposition of the two signals. This often creates a "ghostly" effect where both original sounds are heard simultaneously at varying volumes, rather than a true hybrid sound.

This paper introduces a method that treats audio spectrograms as 2D probability distributions and uses the mathematical framework of Optimal Transport (OT) to find their Wasserstein barycenter. This approach allows for a more natural "movement" of sound energy from the source to the target, preserving the structural and rhythmic characteristics of the signals.

Methodology

Spectrograms as Distributions

The proposed pipeline begins by computing the Short-Time Fourier Transform (STFT) of the source and target sounds. The resulting magnitude spectrograms are normalized and treated as 2D probability distributions over a time-frequency grid. The goal is to find an intermediate distribution—the barycenter—that minimizes the cost of transporting mass from both the source and target.

Structured Cost Matrix and Unbalanced Transport

A key innovation of this work is the design of the transportation cost matrix $\bar{C}$. In standard OT, energy can theoretically move between any two points in the time-frequency plane. However, in audio, moving energy too far across the time axis often destroys the rhythmic structure. The authors introduce a parameter $p$ that penalizes long-distance temporal displacements.

To handle the constraints imposed by this structured cost matrix and to improve computational efficiency, the authors employ Unbalanced Optimal Transport (UOT). Unlike standard OT, which requires strict mass conservation, UOT allows for some mass to be created or destroyed (penalized by Kullback-Leibler divergence), making the optimization more robust for complex audio signals.

Experimental Results

The method was evaluated on both synthetic musical notes (piano to guitar) and environmental sounds (cicada chirp to flowing water). The results demonstrate that the global OT approach, combined with the structured cost matrix, significantly outperforms traditional frame-by-frame or Euclidean methods.

Comparison of interpolation methods for environmental sounds: (a) source cicada chirp, (b) target flowing water, (c) standard OT interpolation, and (d-f) UOT interpolation with varying temporal penalty p.

As shown in the figure above, the parameter $p$ acts as a critical control for the interpolation quality. When $p=0$ (d), the energy movement is restricted to frequency shifts within the same time frame, preserving the rhythmic "envelope" of the cicada chirp while adopting the spectral texture of the flowing water. As $p$ increases (e, f), the temporal structure becomes increasingly blurred, eventually resembling standard OT results where the rhythmic pulses are lost in a continuous drone.

Conclusion and Future Work

By treating spectrograms globally and utilizing Unbalanced Optimal Transport with structured costs, the authors provide a flexible framework for high-quality audio interpolation. While current limitations include the use of the Griffin-Lim algorithm for phase reconstruction and the computational overhead preventing real-time use, the method opens new doors for creative sound design. Future research will explore more advanced phase retrieval techniques and further optimization of the transport solvers.

Summary updated

Abstract

We present a novel approach for generating an artificial audio signal that interpolates between given source and target sounds. Our approach relies on the computation of Wasserstein barycenters of the source and target spectrograms, followed by phase reconstruction and inversion. In contrast with previous works, our new method considers the spectrograms globally and does not operate on a temporal frame-to-frame basis. Another contribution is to endow the transportation cost matrix with a specific structure that prohibits remote displacements of energy along the time axis, and for which optimal transport is made possible by leveraging the unbalanced transport framework. The proposed cost matrix makes sense from the audio perspective and also allows to reduce the computation load. Results with synthetic musical notes and real environmental sounds illustrate the potential of our novel approach.
BibTeX
@inproceedings{valdivia2025audio,
  author = {David Valdivia and Marien Renaud and Elsa Cazelles and Cédric Févotte},
  title = {{Audio signal interpolation using optimal transportation of spectrograms}},
  booktitle = {SSP},
  year = {2025},
  url = {https://openreview.net/forum?id=dgQo2JUv1P},
}
Figure