Autoregressive Visual Modeling via Discrete Tokenization

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2444 papers537 conceptsfrozen 2026-10-04

The meteoric rise of Large Language Models (LLMs) is built on a deceptively simple foundation: next-token prediction. By learning to predict the next word in a sequence, models like GPT-4 develop emergent reasoning and world-modeling capabilities. Researchers are now asking a provocative question: if this "engine" works so well for text, can we treat pixels as a language and train a single model to both "read" and "write" the visual world?

This research direction focuses on visual tokenization—the process of discretizing continuous visual data into a sequence of discrete units that a Transformer can process just like words.

The Image as a Sentence

In traditional computer vision, images are often treated as static grids of pixels or continuous feature vectors. However, as Muhammad Arslan Manzoor and colleagues note in their survey on multimodal evolution, modern systems are bridging the "heterogeneity gap" by discretizing images.

The intuition is to treat an image as a "visual sentence." Just as a sentence is a sequence of tokens from a vocabulary, an image can be decomposed into patches and mapped to a "visual vocabulary" using a Vector-Quantized Variational Autoencoder (VQ-VAE). In early unified models like M6, Junyang Lin and his team demonstrated that by converting images into these discrete codes, a Transformer could learn to generate them autoregressively, predicting the next visual patch based on preceding text and image tokens.

Unifying Understanding and Generation

Historically, AI has separated "understanding" (identifying what is in an image) from "generation" (creating an image). This created a rift between models like CLIP and Stable Diffusion. The autoregressive paradigm heals this rift.

If a model can predict the next visual token, it can perform both tasks. For understanding, the model processes visual tokens and predicts text tokens (captioning). For generation, it processes text tokens and predicts visual tokens (text-to-image). Lijie Fan and the Google DeepMind team explored this in UniFluid, showing that these two tasks actually benefit each other when trained in a unified framework. Similarly, Kaihang Pan introduced Morph-Tokens, which allow tokens to "morph" their roles—acting as abstract concepts for reasoning and high-fidelity blueprints for generation.

From Spatial Patches to Causal Chains

A significant challenge in this field is that images are inherently spatial (2D), while autoregressive models are causal (1D). Simply flattening a grid into a sequence can lose structural context.

Recent work like Selftok by Bohan Wang and Zhongqi Yue proposes a more sophisticated mapping. Instead of spatial patches, they derive tokens from the reverse diffusion process. This converts the image into a "causal chain" of refinement steps. By making visual tokens truly autoregressive, the model can inherit the full power of LLM infrastructure, including Reinforcement Learning (RL). Selftok-Zero applies this to bring the "RLHF moment" to images, allowing models to improve their composition and instruction-following through feedback, much like ChatGPT.

Bridging Modalities with Semantic Tokens

Some researchers argue that visual tokens should be even closer to language. In SweetTokenizer, Zhongquan Tan maps visual features to a "Motion-enhanced Language Codebook" where tokens correspond to semantic text embeddings (like nouns for objects and verbs for motion).

This "visual-as-language" approach is taken to its logical extreme by Xudong Lin, who found that translating video into a sequence of text keywords allows pretrained text models to perform video retrieval with massive efficiency. By forcing visual data into the native embedding space of an LLM, we bypass the need for complex cross-modal fusion layers.

Why the Autoregressive Approach Works

The intuition behind the success of next-token prediction for vision, as summarized in the comprehensive survey by Liang Chen, lies in three areas:

  1. Scaling Laws: Autoregressive models scale predictably with data and compute, a property that has been harder to prove for other architectures.
  2. Unified Reasoning: By using the same causal transformer backbone, the model can use its "reasoning" capabilities (learned from text) to guide its "drawing" capabilities. For example, Janus-4o demonstrates superior instruction-following in image generation because it treats the prompt as a logical sequence to be fulfilled.
  3. Flexibility: Framing vision as a sequence modeling task allows for "in-context learning." Just as you can give an LLM a few examples of a text task, models like Painter by Reza Pourreza can perform new visual tasks (like sketching or editing) simply by seeing a sequence of example "visual tokens."

By treating the visual world as a structured, sequential language, this research direction is moving us toward a truly "general-purpose" multimodal intelligence that can see, reason, and create within a single, unified mathematical framework.

Papers in this version

Showing 60 of 444 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 298 papers
  2. Version 1 · 2026-10-04 · 444 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_discrete_visual_tokenization_vector_quan_f53177ef,
  title = {discrete visual tokenization (vector-quantized image representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-visual-tokenization-vector-quantized-image-representation/rdr_f53177ef20354bc3900f26f946de2108},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_autoregressive_visual_modeling_via_discr_f53177ef_v1,
  title = {Autoregressive Visual Modeling via Discrete Tokenization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/autoregressive-visual-modeling-via-discrete-tokenization/rdr_f53177ef20354bc3900f26f946de2108/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}