Large Language Models (LLMs) owe much of their success to the discrete nature of text. Words can be cleanly mapped to a fixed vocabulary, allowing transformers to predict the next token using standard cross-entropy loss. To replicate this predictable scaling behavior in the visual domain, early generative frameworks like CogView by Chang Zhou et al. introduced Vector Quantization (VQ). By mapping continuous pixel spaces into a finite vocabulary of discrete "visual words," these models successfully applied autoregressive sequence modeling to image synthesis.
However, this translation introduces a fundamental flaw: images are not naturally composed of discrete units. Forcing continuous visual signals—such as subtle color gradients, intricate textures, and sharp geometric edges—into a rigid, finite codebook discards critical high-frequency information. As noted by Arash Mari Oriyad et al. in Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models, this loss of fine-grained detail directly degrades spatial and numerical accuracy, creating a distinct performance gap when compared to continuous diffusion models.
The Mechanics of the Information Bottleneck
To understand why this bottleneck occurs, consider the pipeline of a standard vector-quantized network. An encoder first compresses an image into a continuous latent space. The quantization stage then projects each continuous latent vector to its nearest neighbor within a learned codebook of vectors.
Continuous Image ──> [ Encoder ] ──> Continuous Latent ──> [ Quantization (VQ) ] ──> Discrete Tokens (Lossy)
│
(Discretization Error /
Information Loss)
This nearest-neighbor mapping is inherently lossy. Tianhong Li in Autoregressive Image Generation without Vector Quantization explains that restricting high-dimensional visual data to a finite vocabulary severely limits the model's representational capacity. The downstream transformer can only assemble the discrete tokens it receives; it cannot recover the continuous nuances discarded during the initial quantization phase.
As Huiwen Chang et al. demonstrate with MaskGIT, this creates a hard performance ceiling. If the tokenizer introduces blurriness or structural artifacts during compression, the generative model will inherit and amplify these flaws.
A striking manifestation of this bottleneck is text rendering. In VTBench: Evaluating Visual Tokenizers for Autoregressive Image Generation, Huawei Lin et al. show that while continuous latent models preserve legible text, discrete tokenizers often scramble text into unreadable patterns. Because rendering legible text requires exact, pixel-perfect reconstruction, it serves as a high-fidelity stress test that exposes the severe information loss inherent in discrete quantization.
The Compression vs. Fidelity Dilemma
To keep the sequence lengths manageable for transformers, tokenizers must aggressively compress images (e.g., reducing a $256 \times 256$ image to a $16 \times 16$ grid of tokens). However, higher compression ratios exacerbate the quantization gap. The authors of DC-AR note that applying high compression ratios to discrete tokenizers typically results in severe reconstruction degradation or a failure to generalize across different resolutions.
This creates a direct conflict between the computational efficiency of short token sequences and the visual fidelity required for photorealism. In unified vision-language models, this bottleneck is even more pronounced. Chuofan Ma et al. point out in UniTok: a Unified Tokenizer for Visual Generation and Understanding that a limited discrete token space struggles to simultaneously store low-level visual details and high-level abstract semantics, forcing a trade-off between understanding and generation capabilities.
Escaping the Bottleneck
To overcome this ceiling, researchers are actively developing alternative paradigms that rethink or entirely eliminate discrete quantization:
Hierarchical and Multi-Scale Tokenization
Instead of predicting tokens in a flat, sequential raster scan, models like Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (VAR) by Keyu Tian et al. structure the generation process from coarse to fine scales. By predicting entire resolution scales autoregressively, the model captures global structure at lower resolutions and refines fine details at higher scales, mitigating some of the spatial inaccuracies of flat tokenization.
Continuous Autoregressive Generation
A more radical shift abandons discrete codebooks entirely. Chenze Shao et al. introduce Continuous Visual Autoregressive Generation via Score Maximization, which models continuous-valued tokens directly using strictly proper scoring rules. Similarly, Chunrui Han et al. leverage continuous tokens paired with a lightweight flow-matching head in NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scale, bypassing the quantization bottleneck entirely to achieve the texture fidelity of top-tier diffusion models.
Differentiable Relaxations
Rather than making hard categorical choices during generation, Yuanzhi Zhu et al. propose Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddings, which uses "soft embeddings"—a differentiable relaxation of discrete tokens. This allows the model to optimize expected embedding values end-to-end, mitigating the rigid constraints of traditional vector quantization.