Autoregressive Visual Generation with Continuous Tokens

51 concepts35 papers1 proposal

Large Language Models (LLMs) owe much of their success to the discrete nature of language. Words can be cleanly divided into a finite vocabulary of tokens, allowing transformers to predict the next word using a straightforward classification layer.

When researchers sought to replicate this success in the visual domain, they naturally reached for a similar discretization strategy. Early visual autoregressive models relied on Vector Quantization (VQ)—such as VQ-VAEs—to carve up continuous images into patches and map each patch to an index in a fixed codebook or "visual vocabulary."

However, forcing the continuous, high-dimensional spectrum of the visual world into a finite set of discrete bins creates a severe performance ceiling. Recently, a powerful alternative paradigm has emerged: Continuous Latent Autoregression. By abandoning discrete codebooks and training transformers to predict continuous vectors directly, this approach preserves fine-grained details, stabilizes training, and unlocks true scalability for visual generation.


The Quantization Bottleneck

To understand why continuous tokens work, we must first look at why discrete tokenization fails for images.

In language, the difference between two words is categorical. In vision, the difference between two image patches is a matter of continuous variation—subtle shifts in texture, lighting, and spatial layout. When a VQ model maps an image patch to the closest codebook vector, it draws an arbitrary boundary in a continuous feature space.

This discretization introduces several fundamental flaws:

  1. Information Loss: Distinct visual nuances are conflated if they fall into the same discrete category. As early work by Lavrenko et al. and Yuan et al. pointed out, quantization introduces errors akin to synonymy and polysemy, discarding critical data nuances.
  2. Quantization Artifacts: In generative models, this loss of detail manifests as blocky artifacts in fine anatomical structures or blurry textures, creating a performance ceiling that scaling the model size cannot overcome (Fan et al.).
  3. Training Instabilities: Selecting discrete indices requires non-differentiable operations (like argmax). Bypassing this with straight-through estimators often leads to "codebook collapse," where only a fraction of the vocabulary is utilized, causing optimization to become highly unstable (Gao et al.).

The Core Mechanism: Continuous Tokenization

Continuous latent autoregression bypasses the codebook entirely. Instead of mapping an image patch to a discrete index, the image is mapped into a continuous latent space using a standard, non-quantized Variational Autoencoder (VAE) or a continuous visual encoder (Zhang et al.).

Discrete AR (Traditional):
Image Patch ──> [ Encoder ] ──> [ Vector Quantization ] ──> Discrete Index (e.g., #402) ──> [ Softmax Loss ]

Continuous AR (Modern):
Image Patch ──> [ Encoder ] ──> Continuous Latent Vector (Floating-Point) ──> [ Diffusion / Flow Head ]

By representing image patches as high-dimensional continuous vectors, the model retains raw floating-point precision. This allows the network to preserve high-frequency details—such as text, facial features, and intricate textures—that are typically lost during discretization (Kou et al.; Sun et al.).


Bridging Autoregression and Continuous Spaces

If the tokens are continuous vectors rather than discrete indices, we can no longer use a standard Softmax cross-entropy loss over a fixed vocabulary. How does a transformer predict the "next" continuous token?

Researchers have developed three primary mechanisms to solve this:

1. Density Estimation via GMMs

One approach is to change the prediction head to model a continuous probability distribution. In GIVT (Generative Infinite-Vocabulary Transformers), Tschannen et al. replaced the standard embedding lookup and softmax head with linear projections that predict the parameters of a Gaussian Mixture Model (GMM). This allows the transformer to output continuous vector sequences directly while maintaining standard autoregressive causal masking.

2. Integrated Diffusion and Flow Matching Heads

The most widely adopted strategy combines the sequential structure of autoregressive models with the continuous denoising capabilities of diffusion or flow matching.

Instead of predicting an exact vector, the transformer acts as a conditional backbone. For each step in the sequence, a lightweight head applies a diffusion or flow-matching loss to predict the continuous token value. For example, FlowAR (Ren et al.) and Fluid (Fan et al.) use this hybrid approach to predict continuous latent variables. This design yields smoother training gradients and allows the model to iteratively refine high-frequency textures while leveraging the global structural awareness of the autoregressive transformer.

3. Score Maximization and Energy Transformers

Rather than modeling explicit probabilities or adding diffusion steps, models can be optimized using strictly proper scoring rules. Shao et al. introduced a continuous visual autoregressive framework that utilizes a likelihood-free energy score. This allows the transformer to model continuous image features directly, achieving high synthesis quality with significantly faster inference speeds than multi-step diffusion methods.


Why It Works: Intuition

Continuous latent autoregression succeeds because it combines the best of two worlds: the structural logic of transformers and the mathematical smoothness of continuous spaces.

  • Smooth Optimization Gradients: Because the latent space is fluid and continuous, the loss functions produce smooth, informative gradients throughout the entire training process. This completely eliminates the training instabilities and codebook collapse common to discrete architectures (Gao et al.).
  • Unbounded Vocabulary: A discrete codebook forces a model to choose from a fixed set of visual primitives (e.g., 8,192 words). A continuous token space is effectively an infinite vocabulary, allowing the model to represent an infinite variety of precise spatial layouts and fine details (Tschannen et al.).
  • Contextual Fidelity: Even when predicting a target, keeping the input context un-quantized allows the transformer to maintain high-fidelity textures and coherent global structures across long dependencies (Liu et al.).

Beyond Static Images

The benefits of continuous tokens extend far beyond 2D image generation. The fluid nature of continuous latents makes them natively suited for domains where transitions must be perfectly smooth and unordered data must be handled flexibly:

  • Video Generation: Models like VideoMAR (Yu et al.) represent video frames as continuous vectors, applying a next-frame diffusion loss to maintain temporal causality and spatial fluidity without the massive compute overhead of discrete tokenization.
  • 3D Generation: In MAR-3D, Chen et al. leverage a continuous latent space combined with random masking to accommodate the unordered nature of 3D spatial data, avoiding the structural artifacts that discrete quantization imposes on complex meshes.
  • Human Motion Prediction: Sequential human movement is inherently continuous. By utilizing continuous latent autoregressive modeling, frameworks like WaveAR (Gao et al.) and OmniMotion (Yuan et al.) eliminate the jittery motion artifacts typical of discrete codebooks, preserving the natural fluid nuances of whole-body movement.

By shifting from discrete vocabularies to continuous spaces, modern autoregressive models have broken through the performance ceiling of early visual tokenizers, proving that next-token prediction can match or exceed the fidelity of pure diffusion models at scale (Han et al.).

Key Foundational Work

A Model for Learning the Semantics of Pictures (2003) — contributed 2 concept elements to this direction.
Spatial Random Partition for Common Visual Pattern Discovery (2007) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_latent_space_modeling_596a3288,
  title = {continuous latent space modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-latent-space-modeling/rdr_596a3288259145e2b0eb27c8416c0a5a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_continuous_latent_space_modeling_596a3288_v1,
  title = {continuous latent space modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-latent-space-modeling/rdr_596a3288259145e2b0eb27c8416c0a5a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake