Abstract
In autoregressive (AR) image generation, visual tokenizers compress images into compact discrete latent tokens, enabling efficient training of downstream autoregressive models for visual generation via next-token prediction. While scaling visual tokenizers improves image reconstruction quality, it often degrades downstream generation quality—a challenge not adequately addressed in existing literature. To address this, we introduce GigaTok, the first approach to simultaneously improve image reconstruction, generation, and representation learning when scaling visual tokenizers. We identify the growing complexity of latent space as the key factor behind the reconstruction vs. generation dilemma. To mitigate this, we propose semantic regularization, which aligns tokenizer features with semantically consistent features from a pre-trained visual encoder. This constraint prevents excessive latent space complexity during scaling, yielding consistent improvements in both reconstruction and downstream autoregressive generation. Building on semantic regularization, we explore three key practices for scaling tokenizers: (1) using 1D tokenizers for better scalability, (2) prioritizing decoder scaling when expanding both encoder and decoder, and (3) employing entropy loss to stabilize training for billion-scale tokenizers. By scaling to 3 billion parameters, GigaTok achieves state-of-the-art performance in reconstruction, downstream AR generation, and downstream AR representation quality.
The Reconstruction vs. Generation Dilemma
In the realm of autoregressive image generation, the visual tokenizer acts as the bridge between raw pixels and the discrete tokens that generative models can process. Traditionally, researchers have observed a frustrating paradox: as tokenizers are scaled up to improve image reconstruction (measured by lower rFID), the quality of images generated by downstream models often declines (measured by higher gFID).
This "Reconstruction vs. Generation Dilemma" occurs because larger tokenizers tend to capture excessive low-level noise and fine-grained details. While this helps recreate an exact copy of an input image, it creates a latent space that is too complex and semantically inconsistent for a generator to learn effectively. As shown in Figure 1, baseline models fail to converge or show worsening generation performance as parameters increase, whereas GigaTok maintains a downward trend for both metrics.
Methodology: The GigaTok Approach
To overcome the limitations of standard scaling, GigaTok introduces several architectural and training innovations designed to maintain "learnability" at scale.
Semantic Regularization
The core innovation of GigaTok is semantic regularization. By aligning the tokenizer's latent features with those of a pre-trained, frozen DINOv2 model, the researchers ensure that the tokens represent meaningful semantic concepts rather than just pixel-level textures. This alignment acts as a constraint, preventing the latent space from becoming chaotic as the model grows to billions of parameters.
1D Tokenization and Asymmetric Scaling
Unlike traditional 2D grid-based tokenizers, GigaTok utilizes a 1D sequence approach via a Q-Former. This 1D structure was found to scale more efficiently and remove inductive biases that can hinder performance at massive scales. Furthermore, the authors adopted an asymmetric scaling strategy, allocating more parameters to the decoder. They discovered that the decoder is the primary driver of reconstruction fidelity, while the encoder can remain relatively efficient without sacrificing generative quality.
Training Stability with Entropy Loss
Scaling to 3 billion parameters introduces significant training instability. To combat this, GigaTok employs an entropy loss penalty. This ensures that the "codebook"—the dictionary of available visual tokens—is utilized uniformly, preventing the model from collapsing into a state where only a few tokens are used repeatedly.
Key Results and Insights
GigaTok-XL-XXL represents the largest visual tokenizer to date, reaching 2.9 billion parameters. Its performance sets new benchmarks across several categories:
- State-of-the-Art Reconstruction: It achieves an rFID of 0.79 on ImageNet-1K, the best among discrete tokenizers.
- Superior Generation: When paired with a 1.4B LlamaGen model, GigaTok achieved a gFID of 1.98, outperforming much larger previous SOTA models like VAR and LlamaGen-XXL.
- Efficient Inference: Despite its massive parameter count, the decoding step remains highly efficient, accounting for only about 10% of total inference time in standard AR generation pipelines.
Conclusion and Future Directions
GigaTok demonstrates that scaling visual tokenizers is not only possible but highly beneficial when guided by semantic regularization. By solving the reconstruction vs. generation dilemma, it provides a robust foundation for the next generation of large-scale autoregressive models. Future work will look to expand this framework beyond ImageNet to more complex text-to-image and video generation tasks, as well as exploring the impact of scaling training data to the multi-billion image level.
Summary updated