Taming Audio VAEs via Target-KL Regularization

Prem Seetharaman, Rithesh KumararXiv 2026 · DOI · arXiv

TL;DR

Researchers at Adobe have introduced Target-KL regularization, a novel method for training continuous Variational Autoencoders (VAEs) at specific bitrates. By treating the KL divergence as a controllable "rate" budget, they identified an optimal compression "sweet spot" that significantly improves the performance of downstream latent diffusion models for audio and speech generation.

What Problem Does This Paper Address?

In the world of latent diffusion models (LDMs), the VAE is the unsung hero. It compresses high-dimensional audio into a compact "latent space" that a diffusion model can actually learn to predict. However, training these VAEs is often a "dark art."

There is a fundamental tension:

  1. Over-regularized latents are easy for a diffusion model to learn but result in poor, blurry audio reconstruction.
  2. Under-regularized latents produce near-perfect audio reconstruction but create a "sharp," complex latent space that diffusion models struggle to navigate.

Unlike discrete codecs (like VQ-VAEs), where you can simply set the number of codebooks to control the bitrate, continuous VAEs lack a straightforward way to target a specific compression level. This paper provides the mathematical framework to bridge that gap.

Methods

The authors propose Target-KL regularization. Instead of using a fixed weight ($\lambda$) for the KL divergence term in the loss function—which is notoriously difficult to tune—they introduce a loss that penalizes the model for deviating from a specific Target-KL value:

$$L_{target-KL} = (KL - KL_{target})^2$$

By deriving a relationship between KL divergence, frame rate, and hop length, they can translate a desired bitrate (in kbps) directly into a Target-KL value.

Technically, they build upon the Descript Audio Codec (DAC) architecture but replace the discrete quantization bottleneck with this continuous KL-regularized bottleneck. They also introduced several architectural tweaks to boost quality:

  • Mel-spectrogram projection: Adding mel-spectrogram information directly to the encoder output to speed up convergence.
  • CQT Discriminator: Using a Constant-Q Transform discriminator to better capture musical and harmonic structures.
  • Passthrough Training: Occasionally bypassing the bottleneck entirely during training to help the model learn high-frequency details.

Datasets

  • Training: A large-scale internal proprietary and licensed dataset containing a mix of speech, music, and sound effects.
  • Compression Evaluation: The evaluation subset of AudioSet was used to measure reconstruction error (mel loss).
  • Text-to-Sound Evaluation: The Adobe Audition SFX dataset.
  • Text-to-Speech (TTS) Evaluation: The SEED-en test sets.

Key Insights and Ideas

  • KL as Bitrate: The most important conceptual leap is treating the KL term in the VAE objective as a "coding cost." This allows continuous VAEs to be plotted on the same rate-distortion curves as traditional discrete audio codecs.
  • The Pareto Frontier: The proposed DAC-VAE architecture consistently outperformed existing models like EnCodec, Stable Audio VAE, and SpectroStream, forming the "Pareto frontier"—the best possible quality for any given bitrate.
  • The Generative "Sweet Spot": More data isn't always better. The authors found that for downstream generation, there is an optimal bitrate. If the bitrate is too high, the diffusion model's performance actually drops because the latent space becomes too difficult to model.

Key Results

The authors demonstrated that their method allows for a systematic study of the rate-distortion trade-off, as seen in the curve below:

Fig. 1: Rate-distortion curve for different model families for both discrete and continuous audio compression. With our approach, we can target specific bitrates when training VAEs. We find that our pro- posed DAC-VAE achieves the best audio quality across all bitrates.
  • Text-to-Sound: The best performance (highest text-audio similarity and lowest quality error) was achieved at a bitrate of 11.56 kbps (Target-KL of 200). Increasing the bitrate beyond this point led to worse generative results.
  • Text-to-Speech: Lower bitrates generally led to lower Word Error Rates (WER) and higher speaker similarity (SSIM), suggesting that a smoother, more regularized latent space is crucial for high-quality speech synthesis.
  • Ablation Study: As shown in the table below, adding the CQT discriminator and passthrough training significantly reduced reconstruction error (Mel distance) while maintaining the target bitrate.
Table 1: Ablation study showing the impact of passthrough training and the CQT discriminator on reconstruction quality.

Limitations and Future Work

While the paper provides a robust framework, the authors note a few areas for further exploration:

  • Model Scaling: All models in this study used the same number of parameters. Future work could investigate how increasing the VAE's capacity interacts with the fixed bitrate budget.
  • Semantic Alignment: The authors suggest combining Target-KL with semantic alignment techniques to ensure the latent space captures high-level meaning even more effectively.
  • TTS Complexity: The TTS results showed some intriguing patterns where high-bitrate models had low WER but sounded "monotonous." A deeper investigation into metrics that capture prosody and naturalness is needed.

Summary updated

Abstract

Latent diffusion models have emerged as the dominant paradigm for many generation tasks including audio generation such as text-to-audio, text-to-music and text-to-speech. A key component of latent diffusion is an autoencoder (VAE) that compresses high-dimensional signals into a low frame rate continuous representation that is conducive for downstream prediction. Regularizing these VAEs is challenging, as there is a trade-off between over-regularized (poor output quality) and under-regularized (difficult to predict) latent representations. We propose a framework for studying this trade-off through compression and train Audio VAEs at specific bitrates via target-KL regularization. This allows direct comparison to well-studied discrete neural audio codec models, and the construction of rate-distortion curves for audio VAEs. We evaluate the impact of target-KL regularization on text-to-sound generation and find that sweeping compression rates is helpful in identifying the optimal generation setting.
BibTeX
@misc{seetharaman2026taming,
  author = {Prem Seetharaman and Rithesh Kumar},
  title = {{Taming Audio VAEs via Target-KL Regularization}},
  year = {2026},
  eprint = {2605.17085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.17085},
}
Figure