Discrete Latent Representations for Audio and Speech

26 concepts24 papers1 proposal

Speech and seismic signals are inherently continuous, high-dimensional, and noisy. They contain a mixture of essential information—such as linguistic content or structural seismic patterns—and non-essential variation, like background noise, room reverberation, or individual speaker characteristics. To process these signals effectively, researchers have turned to Vector Quantization (VQ) and Variational Autoencoders (VAEs) to map continuous waveforms into a finite, discrete latent space.

The Intuition of the Information Bottleneck

The core power of discretizing a latent space lies in the "information bottleneck." By forcing a neural network to represent a complex signal using only a small, fixed vocabulary of learnable "codewords," the model is prevented from simply copying the input. Instead, it must learn to discard stochastic noise and focus on the underlying structural patterns.

Liu et al. demonstrate this effect in the context of seismic waveforms, arguing that discretization acts as a robust denoising mechanism. By predicting discrete representations rather than raw values, their model, SeisLM, learns to ignore background noise and capture the fundamental physics of the signal. Similarly, in speech, Baevski et al. found that predicting future time-steps in a quantized latent space (VQ-Wav2Vec) is significantly more effective for discovering phonetic structure than simple reconstruction.

Discovering a Learned Alphabet

When a model like a VQ-VAE is trained on speech, the resulting discrete codes often function as a "learned alphabet" or a set of pseudo-phonemes. This allows for complex speech tasks to be performed without any human-labeled text.

  • Textless Translation: Chiu et al. developed the Textless Translatotron, which translates speech directly into the discrete codes of another language. This bypasses the need for text or phoneme data, making high-quality translation possible for unwritten languages.
  • Acoustic Unit Discovery: Gundogdu et al. utilize vector quantization to discover fundamental acoustic units in low-resource languages, achieving high discrimination accuracy at very low bitrates.
  • Modeling Human Perception: Higy et al. and the authors explore how these vector quantization layers approximate human-like symbolic units, providing a bridge between continuous acoustics and symbolic linguistics.

Disentanglement and Normalization

Discretization also facilitates the separation of different aspects of a signal, such as content, speaker identity, and environment. Because the discrete codes are limited, they tend to capture the most consistent, invariant features (the words being said), while other components (the speaker's voice) are relegated to separate, continuous conditioning variables.

In voice conversion, Choi et al. and Choi et al. use VQ-VAEs to disentangle speaker identity from linguistic content. By conditioning the decoder on a specific speaker code and reverberation information, they can reconstruct speech that retains the original message but adopts a new voice or environment.

Furthermore, Burridge demonstrates that the latent space of a VAE can be used for vowel normalization. By applying transformations directly to the latent features, researchers can normalize speaker-specific variations in a way that mirrors traditional linguistic methods but applies to the entire speech spectrum.

Enhancing Generative Quality

Beyond analysis, discrete representations improve the stability and naturalness of generative models. In non-autoregressive speech synthesis, Chien et al. use a VQ-VAE to compress prosody into a discrete space, making the task of predicting complex pitch and rhythm patterns more manageable for the model.

For high-fidelity tasks, Chen et al. show that performing diffusion in a learned latent space (LDM-SVC) rather than on raw audio leads to superior singing voice conversion with minimal timbre leakage. Similarly, Xie 0001 et al. argue that using a learned latent representation instead of fixed Mel-spectrograms solves the "mismatch problem" in speech synthesis, ensuring that the acoustic model and the vocoder are perfectly aligned.

Key Foundational Work

Hierarchical Prosody Modeling for Non-Autoregressive Speech Synthesis (2020) — contributed 1 concept elements to this direction.
A Comparison of Discrete Latent Variable Models for Speech Representation Learning (2020) — contributed 1 concept elements to this direction.
Vector Quantized Temporally-Aware Correspondence Sparse Autoencoders for Zero-Resource Acoustic Unit Discovery (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_vector_quantization_433949d0,
  title = {vector quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-quantization/rdr_433949d0f3d6424f8408fe446f083364},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_vector_quantization_433949d0_v1,
  title = {vector quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/vector-quantization/rdr_433949d0f3d6424f8408fe446f083364/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake