Traditional classification models treat labels as discrete, mutually exclusive indices in a closed set. If a model is trained on a fixed ontology, it can never output a class outside that predefined list. While contrastive models expand this to an open vocabulary by matching input features against text embeddings, they introduce a new bottleneck: they require a predefined "gallery" or list of candidate labels at inference time. As this gallery scales to millions of concepts, performance degrades and computational costs spike.
A powerful alternative paradigm unifies recognition and labeling across domains by framing the problem not as selection, but as autoregressive text generation. By predicting labels token-by-token over a standard sub-word vocabulary, models can "spell out" what they see or infer. This shift from discriminative classification to generative next-token prediction enables unconstrained, open-vocabulary labeling across vision, language, and source code.
The Core Mechanism: From Softmax to Text Decoders
Instead of a classification head with a fixed number of outputs, generative labeling architectures utilize a language model decoder. Features from the input modality—whether they are image regions, knowledge graph triples, or source code tokens—are projected directly into the embedding space of a language model. The model then autoregressively predicts the next text token until it generates an end-of-sequence marker.
For example, in GIT: A Generative Image-to-text Transformer for Vision and Language, Jianfeng Wang et al. demonstrated that image classification can be effectively framed as a pure text generation task where the model outputs the category name directly string-by-string. This eliminates task-specific output heads and allows the model to handle an open vocabulary of classes without architectural modifications.
Why It Works: Bypassing the Gallery Bottleneck
To understand why this generative approach scales so effectively, consider the computational mechanics of open-world recognition. In a contrastive setup, the model must compute similarity scores between an input embedding and every single candidate label in a provided gallery. If the concept space includes every noun or phrase in the English language, the gallery becomes massive, leading to severe performance degradation.
As argued in Object Recognition as Next Token Prediction by Kaiyu Yue et al., sampling labels directly from a full textual vocabulary avoids this degradation. The language model's vocabulary is fixed (typically 30,000 to 50,000 sub-word tokens), meaning the output layer's computational complexity depends entirely on the length of the generated label string (usually just a few tokens), rather than the size of the open-world search gallery. This allows for truly open-world visual identification, a property also leveraged by Qihang Yu et al. in Towards Open-Ended Visual Recognition with Large Language Models to bypass manual label merging across disparate datasets.
Capturing the Long Tail Through Compositionality
A closed-set classification schema struggles with the "heavy tail" of the real world—fine-grained details, temporary states, and complex interactions that cannot be neatly categorized into a single word. Generative labeling solves this by exploiting the compositional nature of language.
In computer vision, this allows models to move beyond simple object names to rich, localized descriptions:
- Dense Object Understanding: Jialian Wu et al. developed GRiT: A Generative Region-to-text Transformer for Object Understanding, which unifies object detection and dense captioning by letting a decoder spell out object names, attributes, and actions using a sub-word vocabulary. This builds on early foundations laid by Justin Johnson et al. in DenseCap: Fully Convolutional Localization Networks for Dense Captioning.
- Structured Scene Graphs: Instead of classifying relationships from a fixed matrix, Chengyang Zhao et al. showed in TextPSG: Panoptic Scene Graph Generation from Textual Descriptions that treating scene graph labeling as an autoregressive text generation task allows models to leverage linguistic common sense to resolve visual ambiguities and describe novel relationships.
- Complex Interactions: For human-centric scenarios, Boshen Xu et al. in Open-Category Human-Object Interaction Pre-training via Language Modeling Framework and Morris Alper et al. in Learning Human-Human Interactions in Images from Weak Textual Supervision demonstrated that free-form text generation can capture thousands of distinct social behaviors and rare zero-shot interactions by flexibly combining verbs and objects.
- Explainable Computer Vision: Generative labeling can also explain the inner workings of deep networks. In Natural Language Descriptions of Deep Visual Features, Evan Hernandez et al. adapted generative captioning models to synthesize open-ended descriptions of the visual features that maximize an individual neuron's activation, discovering complex concepts that escape fixed dictionaries.
This capacity to handle the long tail is vital for specialized interfaces and tracking systems. For instance, Yang Li et al. applied generative decoding in Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements to capture diverse functional descriptions of UI elements. Similarly, in OCTrack: Benchmarking the Open-Corpus Multi-Object Tracking, Ruize Han et al. removed the dependency on predefined category lists during tracking by projecting visual features into a language space to name objects on the fly.
Extension to Non-Visual Domains: Knowledge Graphs and Code
The intuition that generation beats selection applies far beyond computer vision. Any domain burdened by rigid ontologies or user-defined structures benefits from this paradigm shift.
Structured Knowledge and NLP
In Knowledge Graphs, we often lack simple, human-readable taglines for entities. Rajarshi Bhowmik et al. addressed this in Generating Fine-Grained Open Vocabulary Entity Type Descriptions by using a generative decoder to synthesize fine-grained phrases from an open vocabulary, combining multiple entity attributes without being constrained by a predefined hierarchy. In a similar vein, Leyang Cui et al. introduced Template-Based Named Entity Recognition Using BART, reframing named entity recognition as a language model ranking problem over templates, making the model architecture completely independent of specific entity types.
Generative Type Inference in Source Code
Static analysis of dynamic languages like Python and JavaScript historically relied on classifying variables into a few common types (int, str). However, real-world code contains an infinite tail of custom, user-defined classes and deeply nested parametric structures (e.g., Map<String, List<UserConfig>>).
To solve this, Lukas Seidel et al. in Learning Type Inference for Enhanced Dataflow Analysis and Jiayi Wei et al. in TypeT5: Seq2seq Type Inference using Static Analysis treated type inference as a generative text-infilling or sequence-to-sequence task. Because the model synthesizes structured strings token-by-token rather than selecting from a predefined list, it can predict entirely novel, user-defined data structures and non-local parametric types that were never explicitly present in the training label set.
Interactive Grounding and Distillation
Because the output space of these models is natural language, they can easily bridge the gap between different systems, humans, or other models.
In robotics, Mohit Shridhar et al. developed a "grounding-by-generation" framework in Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction. Instead of matching user commands to a fixed set of objects, the robot generates natural language descriptions for candidate objects and evaluates semantic similarity, allowing it to handle unconstrained vocabulary and proactively ask clarifying questions.
Furthermore, this generative capability can be used to supervise other models. In A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection, the author Shenghao Fu demonstrated that an off-the-shelf generative captioning model can be used for pseudo-labeling. By producing natural language descriptions of unknown image regions and extracting nouns as training labels, the system provides a diverse and open-ended vocabulary to train open-vocabulary object detectors on objects they were never explicitly trained to see. This same generative potential is noted by Zechen Li et al. in SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognition, where moving from fixed-class linear heads to generative outputs is highlighted as the key to unlocking open-set human activity recognition from raw sensor data.
By shifting the paradigm from selecting a label to writing a label, these diverse frameworks leverage the full compositional power and open-ended vocabulary of language models, fundamentally changing how machines recognize the world around them.
Key Foundational Work
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2016) — contributed 1 concept elements to this direction.
Generating Fine-Grained Open Vocabulary Entity Type Descriptions (2018) — contributed 1 concept elements to this direction.
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction (2018) — contributed 1 concept elements to this direction.