Efficient Generative Modeling with Residual Vector Quantization-Based Tokens

Jaehyeon Kim, Taehong Moon, Keon Lee, Jaewoong Cho

TL;DR

ResGen is a novel generative framework that enables high-fidelity image and audio synthesis with remarkably fast sampling speeds. By predicting cumulative vector embeddings instead of individual hierarchical tokens, ResGen decouples inference complexity from quantization depth, allowing it to outperform state-of-the-art autoregressive models in both quality and efficiency.

What Problem Does This Paper Address?

High-fidelity generative modeling often relies on Residual Vector Quantization (RVQ) to compress data into a hierarchical sequence of tokens. While RVQ improves reconstruction quality by adding "depth" (layers of refinement), it creates a bottleneck for generative models: standard autoregressive approaches must predict tokens one by one across both sequence length and depth. This leads to a linear increase in sampling time as fidelity improves.

Existing non-autoregressive methods have tried to tackle this, but they typically fail to eliminate the sampling complexity associated with both dimensions simultaneously. The challenge lies in balancing high-fidelity output with the computational efficiency required for real-time applications like text-to-speech (TTS) or high-resolution image generation.

Methods

ResGen introduces an efficient RVQ-based discrete diffusion process. Its core innovation is Multi-Token Prediction of Masked Tokens.

Instead of predicting discrete token indices at every level of the RVQ hierarchy, ResGen directly predicts the cumulative vector embedding (the sum of the embeddings across depths) for masked positions. This approach is grounded in the observation that the VQ-VAE decoder ultimately operates on these embeddings, making the exact token index less critical than the aggregate vector.

Key components include:

  • Hierarchical Masking Strategy: The model progressively masks tokens starting from the finest details (highest depth) and moving toward coarser information (lower depth).
  • Discrete Diffusion Framework: The authors formulate the masking/unmasking process as a forward and reverse discrete diffusion process, providing a solid probabilistic foundation using variational inference.
  • Mixture of Gaussians (MoG): To handle the continuous nature of the predicted cumulative embeddings, the model employs a MoG distribution, optimized with a specialized loss function to ensure high component usage and diversity.
Figure 1. An overview of the forward masking and reverse unmasking processes is shown at the top, with a detailed depiction of the reverse unmasking process below. In the top figure, forward masking proceeds from right to left, incrementally masking more tokens, while reverse unmasking progresses from left to right, iteratively revealing the masked tokens. White boxes denote masked tokens and colored boxes represent tokens that have been uncovered. The bottom figure illustrates the reverse unmasking process in detail. Starting from masked residual vector quantization (RVQ) tokens, our method first predicts cumulative RVQ embeddings. These embeddings are then quantized and partially masked again. Through a series of iterations, each round predicts the values of the masked tokens and replaces them until the entire token sequence is filled.

Datasets

The authors validated ResGen across two distinct modalities:

  • Vision: Conditional image generation on ImageNet 256×256.
  • Audio: Zero-shot text-to-speech synthesis using two tasks (continuation and cross-sentence) inspired by the Voicebox framework. The models were trained on large-scale speech datasets (referenced as the "Open AI Dataset Project" from AI-Hub).

Key Insights and Ideas

  • Decoupling Depth from Speed: By predicting the sum of embeddings, the number of inference steps becomes independent of the RVQ depth. You can add more layers of refinement to your quantization without slowing down the generator.
  • Vector Space vs. Token Space: Predicting in the embedding space captures correlations among consecutive tokens across depths more effectively than trying to predict individual discrete indices.
  • Efficiency in Parallelism: Unlike the Visual Autoregressive (VAR) model, which requires a fixed resolution hierarchy, ResGen uses a length-agnostic tokenization. This makes it more adaptable to domains with variable lengths, such as audio.
  • Scalability: The model demonstrates "favorable scaling"; as the RVQ depth increases (e.g., from 8 to 16 or even 72 in audio), the generation quality improves significantly with only a negligible impact on inference time.

Key Results

ResGen demonstrates a superior trade-off between generation quality, memory efficiency, and speed.

  • Image Generation: On ImageNet, ResGen-rvq16 achieved an FID of 1.93 (with classifier-free guidance), outperforming the RQ-Transformer and rivaling the MAR series while being significantly faster.
  • Sampling Speed: ResGen ranks as one of the fastest models in its class, second only to VAR, while maintaining higher fidelity than many speed-optimized baselines.
  • Memory Efficiency: It supports the largest latent batch sizes among compared models (e.g., 1,915 for ResGen-rvq16 vs. 1,738 for MAR-B), which is crucial for high-throughput production pipelines.
  • Audio Synthesis: In zero-shot TTS, ResGen achieved the lowest Word Error Rate (WER) and Character Error Rate (CER) in cross-sentence tasks, outperforming models like VALL-E and CLaM-TTS.
Figure 2. The left figure shows the trade-off between sampling speed and generation quality across various generative models. For ResGen, dotted lines indicate performance across different sampling steps, highlighting step-dependent performance improvements. For other models, solid lines connect results corresponding to variations in parameter size. Note that in ResGen-rvq8 and ResGen-rvq16, the number specifies the depth of RVQ. Both wall-clock time and maximum batch size are measured on an NVIDIA A100 GPU.

Limitations and Future Work

While ResGen is highly efficient, the authors note several areas for further exploration:

  • KV Caching: The current transformer architecture does not yet utilize Key-Value (KV) caching, which could further accelerate sampling by reusing computations for already-filled token positions.
  • Quantization Schemes: The method is currently tailored for RVQ. Adapting it to other modern schemes like Finite Scalar Quantization (FSQ) is not straightforward and remains an open research question.
  • Theoretical Gap: Although the model works exceptionally well with a low number of inference steps, a formal theoretical explanation for why so few steps are sufficient (compared to standard Gaussian diffusion) is still needed.

Summary updated

Abstract

We introduce ResGen, an efficient Residual Vector Quantization (RVQ)-based generative model for high-fidelity generation with fast sampling. RVQ improves data fidelity by increasing the number of quantization steps, referred to as depth, but deeper quantization typically increases inference steps in generative models. To address this, ResGen directly predicts the vector embedding of collective tokens rather than individual ones, ensuring that inference steps remain independent of RVQ depth. Additionally, we formulate token masking and multi-token prediction within a probabilistic framework using discrete diffusion and variational inference. We validate the efficacy and generalizability of the proposed method on two challenging tasks across different modalities: conditional image generation on ImageNet 256x256 and zero-shot text-to-speech synthesis. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sampling speeds compared to similarly sized baseline models.
BibTeX
@inproceedings{kim2025efficient,
  author = {Jaehyeon Kim and Taehong Moon and Keon Lee and Jaewoong Cho},
  title = {{Efficient Generative Modeling with Residual Vector Quantization-Based Tokens}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=f8uPzuMNDq},
}
Figure