Morphing Audio through Latent Space Interpolation

18 concepts14 papers1 proposal

Imagine trying to create a sound that is exactly halfway between a violin and a flute. If you take recordings of both instruments and crossfade them using traditional audio editing tools, your brain isn't fooled: you simply hear a violin and a flute playing simultaneously, one fading out while the other fades in. This happens because standard signal processing operates on raw acoustic waveforms or spectral representations, where mixing is purely additive.

To create a true hybrid instrument—a sound that possesses the cohesive, physical characteristics of both parents—we must interpolate not the raw signals, but the underlying generative rules of the audio.

This is the core intuition behind latent space audio interpolation. By training deep generative models to compress audio into a continuous, low-dimensional latent space, researchers have discovered that the paths between distinct sounds map to smooth, structurally sound transformations. Traversing these spaces allows us to morph timbres, musical styles, and acoustic environments in ways that traditional signal processing cannot replicate.


Why Latent Spaces Enable Structural Morphing

When an autoencoder is trained on a vast dataset of audio, it learns to discard redundant phase and waveform data, compressing the signal into a compact vector called a latent embedding. Because the decoder must reconstruct realistic audio from any valid point in this space, the model naturally organizes the latent space according to the fundamental features of the data, such as pitch, timbre, and dynamics.

[Raw Audio: Violin] ----> ( Encoder ) ----> [ Latent Vector A ]
                                                   |
                                         ( Linear Interpolation )
                                                   |
                                                   v
[Hybrid Audio]       <---- ( Decoder ) <---- [ Latent Vector C ]
                                                   ^
                                                   |
                                         ( Linear Interpolation )
                                                   |
[Raw Audio: Flute]  ----> ( Encoder ) ----> [ Latent Vector B ]

In their seminal work on the NSynth dataset, Jesse Engel et al. demonstrated this property using a WaveNet-based autoencoder in Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. By interpolating between the latent embeddings of two different instruments, the authors synthesized novel, hybrid timbres that maintained realistic acoustic properties throughout the entire transition. Instead of a ghostly overlay of two signals, the model generated a single, evolving sound that adopted the acoustic qualities of both source instruments.

This capability stems from the fact that the latent space captures a continuous manifold of valid audio features. As Ke Chen et al. observed in MusicLDM: Enhancing Novelty in text-to-music Generation Using Beat-Synchronous mixup Strategies, interpolating within a compressed latent space leverages the underlying musical rules learned by the network. This ensures that intermediate samples reside on a valid manifold, producing significantly more coherent and diverse synthetic outputs than mixing raw waveforms. Recent architectures, such as the transformer-based synthesizer proposed by Junghyun Koo et al. in TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument, confirm that this underlying representation remains continuous and highly controllable even when scaling to discrete token spaces.


The utility of latent interpolation extends far beyond simple timbral morphing; it can be applied to complex, multi-dimensional musical structures like harmony, performance style, and environmental acoustics.

Harmonic Trajectories

Instead of dealing with discrete note events, models can treat musical structures as continuous paths. In Latent Chords: Generative Piano Chord Synthesis with Variational Autoencoders, Agustín Macaya et al. mapped piano chords to a two-dimensional latent space. By sampling coordinates along trajectories within this space, the system synthesizes "in-between" musical states, enabling fluid harmonic transitions that do not exist in traditional music theory. Similarly, Gino Brunner et al. developed MIDI-VAE: Modeling Dynamics and Instrumentation of Music with Applications to Style Transfer, demonstrating that interpolating between distinct musical pieces allows a model to morph pitch, dynamics, and instrumentation simultaneously, capturing a coherent joint distribution of multiple musical attributes.

Style and Structure

When applied to broader compositions, latent transformations can blend entirely disparate tracks. Abudukelimu Wuerkaixi et al. showed in CollageNet: Fusing arbitrary melody and accompaniment into a coherent song that manipulating musical segments within a compressed latent space yields significantly higher harmony rates than rule-based methods. This semantic smoothness is also evident in performance characteristics; Kristy Choi et al. demonstrated in Encoding Musical Style with Transformer Autoencoders that traversing the latent space between two performance styles allows a model to generate music that transitions fluidly in expressive attributes like volume and tempo.

Environmental Acoustics

The same interpolation mechanics apply to the physical spaces in which sound occurs. In ReverbMiipher: Generative Speech Restoration meets Reverberation Characteristics Controllability, Wataru Nakata et al. isolated acoustic environments within a continuous latent space. By linearly interpolating between embeddings, users can smoothly transition a speech signal from an anechoic chamber to a highly reverberant hall, providing fine-grained control over the perceived physical environment.


The Challenge of Perceptual Linearity

While the mathematics of moving from point $A$ to point $B$ in a latent space is straightforward—usually a simple linear interpolation ($A \times (1-t) + B \times t$)—the human perception of that transition is rarely linear.

As Satvik Dixit et al. pointed out in Learning Perceptually Relevant Temporal Envelope Morphing, standard linear interpolation in latent spaces often fails to produce perceptually natural transitions. Instead of finding a structural midpoint, the model may still generate an overlapping or "ghostly" effect because the network's internal geometry does not perfectly mirror human psychoacoustics. True perceptual intermediacy requires the model to alter structural elements—such as the number of acoustic events or rhythmic density—rather than simply averaging feature values.

To solve this, Xinlei Niu et al. introduced a framework called SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model. They proposed a metric called Sound Perceptual Distance Proportion (SPDP) and utilized a binary search algorithm to navigate the latent space of a diffusion model. Instead of moving in fixed mathematical increments, the system adjusts its step size based on human auditory perception, ensuring that the morphing trajectory feels perfectly uniform to a listener.

Mathematical Linear Steps:   |----|----|----|----|----|----|
Perceptual Progress:         |--|---|-----|-------|----------| (Feels uneven)

Perceptually Uniform Steps:  |--|---|-----|-------|----------| (Adjusted steps)
Perceptual Progress:         |----|----|----|----|----|----| (Feels smooth)

Alternative Manifolds and Applications

To bypass the unpredictability of standard deep learning latent spaces, some researchers explore alternative manifold designs. For instance, Mathieu Andreux introduced Music Generation and Transformation with Moment Matching-Scattering Inverse Networks, which replaces learned encoders with a fixed, mathematically-defined scattering transform. This explicitly linearizes specific musical attributes, making operations like pitch interpolation highly predictable. From a biological perspective, Dmitry N. Zotkin et al. demonstrated in Neuromimetic Sound Representation for Percept Detection and Manipulation that performing geometric averaging on high-dimensional "cortical representations" (mimicking mammalian auditory processing) provides a naturally linear-like space for interpolating complex auditory textures.

The practical applications of these continuous spaces are widespread:

By shifting our approach from manipulating raw sound waves to navigating continuous, structurally aware latent manifolds, we gain the ability to treat audio as a malleable, fluid medium. The power of the idea lies in its capacity to respect the intrinsic rules of sound, giving us the tools to synthesize entirely new acoustic realities that bridge the gaps between the sounds we already know.

Key Foundational Work

Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders (2017) — contributed 2 concept elements to this direction.
Neuromimetic Sound Representation for Percept Detection and Manipulation (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_space_interpolation_latent_space__5f9f5ed4,
  title = {latent space interpolation (latent space audio interpolation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-interpolation-latent-space-audio-interpolation/rdr_5f9f5ed46bcf4046b806ff437d375a16},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_space_interpolation_latent_space__5f9f5ed4_v1,
  title = {latent space interpolation (latent space audio interpolation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-interpolation-latent-space-audio-interpolation/rdr_5f9f5ed46bcf4046b806ff437d375a16/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake