Generative Object Recognition via Next Token Prediction

This direction is merged into another direction: open-vocabulary label generation. Its last version stays readable and citable here.

Version 1 of 111 papers16 conceptsfrozen 2026-10-04

For years, computer vision has treated object recognition as a "multiple-choice" test. Whether through a fixed-category classifier or a contrastive model like CLIP that matches images to a predefined gallery of text, the model is always choosing from a list.

A new wave of research is reimagining this task as a "free-response" question. Instead of picking a label, models are now "spelling out" what they see, token by token. By framing object recognition as next-token prediction, researchers are merging the architectural strengths of Large Language Models (LLMs) with visual perception to create more flexible, open-vocabulary systems.

From Retrieval to Generation

Traditional open-vocabulary models rely on similarity matching. To identify an object, they compare a visual embedding against a "gallery" of candidate text embeddings. While effective, this approach has a scaling problem: as the number of possible categories grows into the thousands or millions, the computational cost of searching the gallery becomes a bottleneck.

In contrast, generative models like GRiT, developed by Jialian Wu and colleagues, treat the object’s name as a sequence of sub-word tokens. Instead of searching a list, the model uses a transformer decoder to autoregressively generate the text. This shift from discriminative classification to generative decoding allows the model to handle an effectively infinite vocabulary without the overhead of a massive retrieval gallery.

The Mechanism of Naming

The core intuition behind this direction is that a language decoder can serve as a universal interface for visual concepts. In the framework proposed by Kaiyu Yue, Jonas Geiping, and Tom Goldstein in Object Recognition as Next Token Prediction, visual features are treated as the "prefix" or prompt for a language model.

The process typically follows three steps:

  1. Visual Encoding: An encoder (like a ViT) extracts features from an image or a specific region.
  2. Feature Projection: These visual features are projected into the same embedding space used by a language model.
  3. Autoregressive Decoding: A language head (often a truncated version of a model like LLaMA or OPT) predicts the most likely next token to describe that visual input.

By using a "non-causal" attention mask during this process, Yue et al. demonstrate that these models can be up to 18x faster than standard generative approaches while remaining more accurate than retrieval-based counterparts.

Why Generation Beats Selection

The power of this approach lies in its flexibility. Because the model is not tethered to a fixed list of labels, it can describe the world with much higher granularity.

  • Open-Corpus Tracking: In the realm of video, Ruize Han and Wei Feng introduced OCTrack, which uses a generative head to name tracked objects on the fly. This eliminates the need for a predefined category list, allowing the tracker to identify rare or specific objects that weren't in the training set.
  • Dense Segmentation: Models like ROSE by Y Hu and Yunchao Wei apply this to every patch in an image. By decoding patch-wise embeddings into free-form text, they can segment and name every object in a scene autonomously.
  • Scene Understanding: TextPSG by Chengyang Zhao and Chuang Gan extends this to relationships. Instead of just identifying "man" and "bike," the model can generate a full scene graph by "speaking" the relationship "man riding bike" directly from the visual features.

A Unified Vision-Language Architecture

Perhaps the most significant advantage of this paradigm is unification. Historically, object detection, classification, and captioning required different heads and loss functions.

By treating everything as text generation, researchers can use a single architecture for multiple tasks. As seen in GenerateU, the model doesn't just detect a box; it generates the name of the object inside it. This blurs the line between "detecting" an object and "describing" it. If a model can generate a label, it can just as easily generate a caption or a detailed attribute list, all through the same next-token prediction mechanism.

This shift suggests that the future of computer vision may not lie in better classifiers, but in better "talkers"—models that can look at a region of pixels and fluently describe exactly what they see.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_object_recognition_via_next_t_0fd9e233,
  title = {Generative Object Recognition via Next Token Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-object-recognition-via-next-token-prediction/rdr_0fd9e23341c04e2ab0c3db936126e945},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_object_recognition_via_next_t_0fd9e233_v1,
  title = {Generative Object Recognition via Next Token Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-object-recognition-via-next-token-prediction/rdr_0fd9e23341c04e2ab0c3db936126e945/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}