TL;DR
ResGen is a novel generative framework that enables high-fidelity image and audio synthesis with remarkably fast sampling speeds. By predicting cumulative vector embeddings instead of individual hierarchical tokens, ResGen decouples inference complexity from quantization depth, allowing it to outperform state-of-the-art autoregressive models in both quality and efficiency.
What Problem Does This Paper Address?
High-fidelity generative modeling often relies on Residual Vector Quantization (RVQ) to compress data into a hierarchical sequence of tokens. While RVQ improves reconstruction quality by adding "depth" (layers of refinement), it creates a bottleneck for generative models: standard autoregressive approaches must predict tokens one by one across both sequence length and depth. This leads to a linear increase in sampling time as fidelity improves.
Existing non-autoregressive methods have tried to tackle this, but they typically fail to eliminate the sampling complexity associated with both dimensions simultaneously. The challenge lies in balancing high-fidelity output with the computational efficiency required for real-time applications like text-to-speech (TTS) or high-resolution image generation.
Methods
ResGen introduces an efficient RVQ-based discrete diffusion process. Its core innovation is Multi-Token Prediction of Masked Tokens.
Instead of predicting discrete token indices at every level of the RVQ hierarchy, ResGen directly predicts the cumulative vector embedding (the sum of the embeddings across depths) for masked positions. This approach is grounded in the observation that the VQ-VAE decoder ultimately operates on these embeddings, making the exact token index less critical than the aggregate vector.
Key components include:
- Hierarchical Masking Strategy: The model progressively masks tokens starting from the finest details (highest depth) and moving toward coarser information (lower depth).
- Discrete Diffusion Framework: The authors formulate the masking/unmasking process as a forward and reverse discrete diffusion process, providing a solid probabilistic foundation using variational inference.
- Mixture of Gaussians (MoG): To handle the continuous nature of the predicted cumulative embeddings, the model employs a MoG distribution, optimized with a specialized loss function to ensure high component usage and diversity.
Datasets
The authors validated ResGen across two distinct modalities:
- Vision: Conditional image generation on ImageNet 256×256.
- Audio: Zero-shot text-to-speech synthesis using two tasks (continuation and cross-sentence) inspired by the Voicebox framework. The models were trained on large-scale speech datasets (referenced as the "Open AI Dataset Project" from AI-Hub).
Key Insights and Ideas
- Decoupling Depth from Speed: By predicting the sum of embeddings, the number of inference steps becomes independent of the RVQ depth. You can add more layers of refinement to your quantization without slowing down the generator.
- Vector Space vs. Token Space: Predicting in the embedding space captures correlations among consecutive tokens across depths more effectively than trying to predict individual discrete indices.
- Efficiency in Parallelism: Unlike the Visual Autoregressive (VAR) model, which requires a fixed resolution hierarchy, ResGen uses a length-agnostic tokenization. This makes it more adaptable to domains with variable lengths, such as audio.
- Scalability: The model demonstrates "favorable scaling"; as the RVQ depth increases (e.g., from 8 to 16 or even 72 in audio), the generation quality improves significantly with only a negligible impact on inference time.
Key Results
ResGen demonstrates a superior trade-off between generation quality, memory efficiency, and speed.
- Image Generation: On ImageNet, ResGen-rvq16 achieved an FID of 1.93 (with classifier-free guidance), outperforming the RQ-Transformer and rivaling the MAR series while being significantly faster.
- Sampling Speed: ResGen ranks as one of the fastest models in its class, second only to VAR, while maintaining higher fidelity than many speed-optimized baselines.
- Memory Efficiency: It supports the largest latent batch sizes among compared models (e.g., 1,915 for ResGen-rvq16 vs. 1,738 for MAR-B), which is crucial for high-throughput production pipelines.
- Audio Synthesis: In zero-shot TTS, ResGen achieved the lowest Word Error Rate (WER) and Character Error Rate (CER) in cross-sentence tasks, outperforming models like VALL-E and CLaM-TTS.
Limitations and Future Work
While ResGen is highly efficient, the authors note several areas for further exploration:
- KV Caching: The current transformer architecture does not yet utilize Key-Value (KV) caching, which could further accelerate sampling by reusing computations for already-filled token positions.
- Quantization Schemes: The method is currently tailored for RVQ. Adapting it to other modern schemes like Finite Scalar Quantization (FSQ) is not straightforward and remains an open research question.
- Theoretical Gap: Although the model works exceptionally well with a low number of inference steps, a formal theoretical explanation for why so few steps are sufficient (compared to standard Gaussian diffusion) is still needed.
Summary updated