Open Vocabulary Labeling via Next Token Prediction

31 concepts22 papers2 proposals

Traditional classification models treat labels as discrete, mutually exclusive indices in a closed set. If a model is trained on a fixed ontology, it can never output a class outside that predefined list. While contrastive models expand this to an open vocabulary by matching input features against text embeddings, they introduce a new bottleneck: they require a predefined "gallery" or list of candidate labels at inference time. As this gallery scales to millions of concepts, performance degrades and computational costs spike.

A powerful alternative paradigm unifies recognition and labeling across domains by framing the problem not as selection, but as autoregressive text generation. By predicting labels token-by-token over a standard sub-word vocabulary, models can "spell out" what they see or infer. This shift from discriminative classification to generative next-token prediction enables unconstrained, open-vocabulary labeling across vision, language, and source code.

The Core Mechanism: From Softmax to Text Decoders

Instead of a classification head with a fixed number of outputs, generative labeling architectures utilize a language model decoder. Features from the input modality—whether they are image regions, knowledge graph triples, or source code tokens—are projected directly into the embedding space of a language model. The model then autoregressively predicts the next text token until it generates an end-of-sequence marker.

For example, in GIT: A Generative Image-to-text Transformer for Vision and Language, Jianfeng Wang et al. demonstrated that image classification can be effectively framed as a pure text generation task where the model outputs the category name directly string-by-string. This eliminates task-specific output heads and allows the model to handle an open vocabulary of classes without architectural modifications.

To understand why this generative approach scales so effectively, consider the computational mechanics of open-world recognition. In a contrastive setup, the model must compute similarity scores between an input embedding and every single candidate label in a provided gallery. If the concept space includes every noun or phrase in the English language, the gallery becomes massive, leading to severe performance degradation.

As argued in Object Recognition as Next Token Prediction by Kaiyu Yue et al., sampling labels directly from a full textual vocabulary avoids this degradation. The language model's vocabulary is fixed (typically 30,000 to 50,000 sub-word tokens), meaning the output layer's computational complexity depends entirely on the length of the generated label string (usually just a few tokens), rather than the size of the open-world search gallery. This allows for truly open-world visual identification, a property also leveraged by Qihang Yu et al. in Towards Open-Ended Visual Recognition with Large Language Models to bypass manual label merging across disparate datasets.

Capturing the Long Tail Through Compositionality

A closed-set classification schema struggles with the "heavy tail" of the real world—fine-grained details, temporary states, and complex interactions that cannot be neatly categorized into a single word. Generative labeling solves this by exploiting the compositional nature of language.

In computer vision, this allows models to move beyond simple object names to rich, localized descriptions:

This capacity to handle the long tail is vital for specialized interfaces and tracking systems. For instance, Yang Li et al. applied generative decoding in Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements to capture diverse functional descriptions of UI elements. Similarly, in OCTrack: Benchmarking the Open-Corpus Multi-Object Tracking, Ruize Han et al. removed the dependency on predefined category lists during tracking by projecting visual features into a language space to name objects on the fly.

Extension to Non-Visual Domains: Knowledge Graphs and Code

The intuition that generation beats selection applies far beyond computer vision. Any domain burdened by rigid ontologies or user-defined structures benefits from this paradigm shift.

Structured Knowledge and NLP

In Knowledge Graphs, we often lack simple, human-readable taglines for entities. Rajarshi Bhowmik et al. addressed this in Generating Fine-Grained Open Vocabulary Entity Type Descriptions by using a generative decoder to synthesize fine-grained phrases from an open vocabulary, combining multiple entity attributes without being constrained by a predefined hierarchy. In a similar vein, Leyang Cui et al. introduced Template-Based Named Entity Recognition Using BART, reframing named entity recognition as a language model ranking problem over templates, making the model architecture completely independent of specific entity types.

Generative Type Inference in Source Code

Static analysis of dynamic languages like Python and JavaScript historically relied on classifying variables into a few common types (int, str). However, real-world code contains an infinite tail of custom, user-defined classes and deeply nested parametric structures (e.g., Map<String, List<UserConfig>>).

To solve this, Lukas Seidel et al. in Learning Type Inference for Enhanced Dataflow Analysis and Jiayi Wei et al. in TypeT5: Seq2seq Type Inference using Static Analysis treated type inference as a generative text-infilling or sequence-to-sequence task. Because the model synthesizes structured strings token-by-token rather than selecting from a predefined list, it can predict entirely novel, user-defined data structures and non-local parametric types that were never explicitly present in the training label set.

Interactive Grounding and Distillation

Because the output space of these models is natural language, they can easily bridge the gap between different systems, humans, or other models.

In robotics, Mohit Shridhar et al. developed a "grounding-by-generation" framework in Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction. Instead of matching user commands to a fixed set of objects, the robot generates natural language descriptions for candidate objects and evaluates semantic similarity, allowing it to handle unconstrained vocabulary and proactively ask clarifying questions.

Furthermore, this generative capability can be used to supervise other models. In A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection, the author Shenghao Fu demonstrated that an off-the-shelf generative captioning model can be used for pseudo-labeling. By producing natural language descriptions of unknown image regions and extracting nouns as training labels, the system provides a diverse and open-ended vocabulary to train open-vocabulary object detectors on objects they were never explicitly trained to see. This same generative potential is noted by Zechen Li et al. in SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition, where moving from fixed-class linear heads to generative outputs is highlighted as the key to unlocking open-set human activity recognition from raw sensor data.

By shifting the paradigm from selecting a label to writing a label, these diverse frameworks leverage the full compositional power and open-ended vocabulary of language models, fundamentally changing how machines recognize the world around them.

Key Foundational Work

DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2016) — contributed 1 concept elements to this direction.
Generating Fine-Grained Open Vocabulary Entity Type Descriptions (2018) — contributed 1 concept elements to this direction.
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_open_vocabulary_label_generation_f70df239,
  title = {open-vocabulary label generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/open-vocabulary-label-generation/rdr_f70df23943e046ce86fddbf6781bcf03},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_open_vocabulary_label_generation_f70df239_v1,
  title = {open-vocabulary label generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/open-vocabulary-label-generation/rdr_f70df23943e046ce86fddbf6781bcf03/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake