Efficient Generative Modeling with Residual Vector Quantization-Based Tokens

Jaehyeon Kim, Taehong Moon, Keon Lee, Jaewoong Cho

Abstract

We explore the use of Residual Vector Quantization (RVQ) for high-fidelity generation in vector-quantized generative models. This quantization technique maintains higher data fidelity by employing more in-depth tokens. However, increasing the token number in generative models leads to slower inference speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete diffusion model that generates high-fidelity samples without compromising sampling speed. Our key idea is a direct prediction of vector embedding of collective tokens rather than individual ones. Moreover, we demonstrate that our proposed token masking and multi-token prediction method can be formulated within a principled probabilistic framework using a discrete diffusion process and variational inference. We validate the efficacy and generalizability of the proposed method on two challenging tasks across different modalities: conditional image generation on ImageNet 256x256 and zero-shot text-to-speech synthesis. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sampling speeds compared to similarly sized baseline models.

Introduction

In the landscape of generative AI, continuous data such as images or audio are typically converted into discrete "tokens" using Vector Quantization (VQ). To achieve high-fidelity results, researchers often employ Residual Vector Quantization (RVQ), which represents data as a hierarchy of tokens across multiple "depths." While increasing depth improves quality, it traditionally introduces a significant bottleneck: slower inference. Standard autoregressive models must predict tokens sequentially, and with RVQ, the sequence length effectively multiplies by the depth. ResGen is designed to break this "depth-speed" trade-off by decoupling sampling speed from quantization depth.

Methods: Multi-Token Prediction via Cumulative Embeddings

The core innovation of ResGen is the direct prediction of the cumulative vector embedding ($z_i$) for all tokens at a specific spatial position, rather than attempting to predict every individual token at every depth level ($x_{i,j}$).

Overview of the ResGen architecture and the reverse unmasking process.

Key technical components of the ResGen framework include:

  • Masked Token Modeling (MTM): ResGen utilizes a discrete diffusion-like process that begins with a fully masked sequence and iteratively "unmasks" tokens to reveal the final output.
  • Probabilistic Framework: The unmasking process is grounded in variational inference and discrete diffusion, providing a solid theoretical foundation for the sampling strategy.
  • Mixture of Gaussians (MoG) with Low-Rank Projections: To handle the prediction of continuous vector embeddings efficiently, the model uses an MoG head. Low-rank projections are employed to prevent the output layer size from exploding as mixture components increase.
  • Coarse-to-Fine Masking: The masking strategy is specifically designed to reveal coarser, low-depth information before filling in finer details at higher depths.

Datasets and Evaluation

The authors validated ResGen across two distinct modalities to demonstrate its generalizability:

  • Vision: Class-conditional image generation on ImageNet 256×256.
  • Audio: Zero-shot text-to-speech (TTS) synthesis using tasks inspired by the Voicebox framework. The model utilized a pre-trained MelVAE module for underlying audio representation.

Key Insights

  • Decoupling Depth from Speed: By predicting aggregate vectors, the number of inference steps becomes independent of the RVQ depth. This allows for higher "resolution" quantization without increasing the computational time required for sampling.
  • The Efficiency of Unmasking: The authors hypothesize that unmasking (predicting from partially revealed data) is inherently easier than denoising (predicting from pure noise), allowing ResGen to reach high quality in very few iterations.
  • Cross-Depth Correlation: Predicting cumulative embeddings naturally captures the correlations between tokens at different depths, a feature often lost in models that treat depths as strictly sequential or independent.

Key Results

ResGen demonstrates an impressive balance of quality, speed, and memory efficiency compared to existing state-of-the-art models.

Comparison of FID vs. Wallclock Time and Maximum batch size across different models.
  • Image Quality: On ImageNet, ResGen-rvq16 achieved a Fréchet Inception Distance (FID) of 1.93 (with classifier-free guidance). This is significantly better than the autoregressive RQ-Transformer (3.80) and competitive with top-tier models like MAR-L (1.78).
  • Sampling Speed: ResGen is roughly 1.8x to 3.4x faster than its autoregressive counterparts at comparable quality levels.
  • Memory Efficiency: It supports a maximum latent batch size of 1,915 on an NVIDIA A100, outperforming the memory efficiency of the MAR-B model (1,738).
  • Audio Performance: In zero-shot TTS, ResGen achieved the lowest Word Error Rate (WER) in cross-sentence tasks, requiring only 25 iterations to generate audio from a complex 72-level RVQ hierarchy.

Limitations and Future Work

While ResGen represents a significant advancement, the authors identify several areas for future exploration. The current implementation does not yet utilize KV caching, which could further accelerate sampling. Additionally, adapting the model to alternative quantization schemes like Finite Scalar Quantization (FSQ) remains an open challenge. Finally, while empirical results are strong, a formal analytic explanation for why so few inference steps are sufficient for high-quality generation is still being researched.

Summary updated

Abstract

We explore the use of Residual Vector Quantization (RVQ) for high-fidelity generation in vector-quantized generative models. This quantization technique maintains higher data fidelity by employing more in-depth tokens. However, increasing the token number in generative models leads to slower inference speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete diffusion model that generates high-fidelity samples without compromising sampling speed. Our key idea is a direct prediction of vector embedding of collective tokens rather than individual ones. Moreover, we demonstrate that our proposed token masking and multi-token prediction method can be formulated within a principled probabilistic framework using a discrete diffusion process and variational inference. We validate the efficacy and generalizability of the proposed method on two challenging tasks across different modalities: conditional image generation} on ImageNet 256x256 and zero-shot text-to-speech synthesis. Experimental results demonstrate that ResGen outperforms autoregressive counterparts in both tasks, delivering superior performance without compromising sampling speed. Furthermore, as we scale the depth of RVQ, our generative models exhibit enhanced generation fidelity or faster sampling speeds compared to similarly sized baseline models. The project page can be found at https://resgen-genai.github.io
BibTeX
@inproceedings{kim2025efficient,
  author = {Jaehyeon Kim and Taehong Moon and Keon Lee and Jaewoong Cho},
  title = {{Efficient Generative Modeling with Residual Vector Quantization-Based Tokens}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=f8uPzuMNDq},
}
Figure