Introduction
Deep generative models often represent continuous data, such as images or audio, as a sequence of discrete tokens using Vector Quantization (VQ). While standard VQ-VAEs are common, they frequently suffer from poor reconstruction quality. Residual Vector Quantization (RVQ) addresses this by iteratively quantizing the residuals of previous steps, creating a "hierarchy" of tokens that capture increasingly fine details.
However, this hierarchy introduces a significant computational cost: the total number of tokens to predict becomes the product of the sequence length and the quantization depth. For autoregressive models, this leads to agonizingly slow sampling. Even non-autoregressive models struggle to handle both dimensions efficiently, creating a "sampling complexity" gap where increasing data fidelity (via deeper RVQ) traditionally forces a trade-off with generation speed.
ResGen is a novel generative framework designed to solve this efficiency bottleneck. By directly predicting cumulative vector embeddings instead of individual discrete tokens, ResGen decouples the number of inference steps from the quantization depth, enabling high-fidelity synthesis with significantly faster sampling and lower memory overhead.
Methodology
The core innovation of ResGen is a shift from predicting discrete token indices to predicting cumulative vector embeddings.
- Direct Vector Prediction: Instead of predicting which specific token belongs at each level of the RVQ hierarchy, ResGen predicts the sum of the vector embeddings across those levels ($z_i = \sum e_{i,j}$). This allows the model to capture correlations across the entire depth dimension in a single step.
- Coarse-to-Fine Masking: The authors introduce a masking strategy tailored for RVQ. They progressively mask tokens starting from the highest quantization depths (which contain the finest details) down to the lowest (coarser information). This ensures the model learns to refine coarse structures into high-fidelity outputs.
- Probabilistic Framework: The process is formulated as a discrete diffusion process. During training, the model learns to maximize the log-likelihood of the sum of masked embeddings. During inference, it uses an iterative unmasking process (similar to MaskGIT) but operates on the aggregated vectors.
- Mixture of Gaussians (MoG): To model the distribution of these cumulative embeddings, the model outputs parameters for a Mixture of Gaussians, which is more expressive than simple point estimation.
Experimental Setup
The authors evaluate ResGen on two distinct modalities to prove its generalizability:
- Vision: Conditional image generation on ImageNet 256×256.
- Audio: Zero-shot text-to-speech (TTS) synthesis using the LibriSpeech dataset and other proprietary high-quality speech data. Tasks include "continuation" (extending a 3-second clip) and "cross-sentence" (transferring style to new text).
Key Insights and Results
ResGen demonstrates a superior balance of speed, memory usage, and quality by decoupling depth from complexity. Because the model predicts vectors rather than performing massive softmax operations over huge vocabularies at every depth, it is extremely memory-efficient.
- Image Generation: On ImageNet, ResGen-rvq16 achieves an FID of 1.93 (with Classifier-Free Guidance), outperforming the RQ-Transformer (3.80) and DiT-XL/2 (2.27).
- Speed and Throughput: ResGen ranks second only to VAR in raw speed but supports a massive maximum batch size of 1,915 on an NVIDIA A100—significantly higher than MAR-H (812) or VAR-d30 (60).
- Audio Synthesis: In zero-shot TTS, ResGen achieves the lowest Word Error Rate (WER) and Character Error Rate (CER) in cross-sentence tasks, outperforming established baselines like Voicebox and CLaM-TTS while requiring only 25 inference steps for a 72-layer RVQ hierarchy.
The authors suggest a "Simpler Prediction" hypothesis to explain this performance: predicting tokens based on completely unmasked positions (the masking approach) is inherently easier than predicting them from noisy inputs (the standard diffusion approach), allowing ResGen to achieve high quality in very few iterations.
Limitations and Future Work
While highly effective, ResGen has a few notable limitations:
- KV Caching: The current implementation does not utilize Key-Value (KV) caching in the transformer architecture. Implementing this could further reduce computational overhead during the iterative unmasking process.
- Alternative Quantization: The framework is specifically designed for RVQ. Adapting it to other popular methods like Finite Scalar Quantization (FSQ) is not straightforward and remains an open research question.
- Theoretical Grounding: While empirically successful, the authors note a lack of a formal analytic explanation for why such a low number of inference steps (e.g., 25 steps for 72 layers) is sufficient to produce high-quality results.
Summary updated