Introduction
Traditional object recognition has long been divided into two distinct paradigms. Closed-set classification models, such as ResNet, rely on fixed linear layers to predict a predefined set of categories, making them incapable of recognizing concepts outside their training set. Conversely, retrieval-based models like CLIP match image embeddings against a gallery of text descriptions. While more flexible, retrieval-based methods are constrained by the quality and breadth of the predefined gallery; if a concept is missing from the list, the model remains "blind" to it.
Generative multimodal models (e.g., LLaVA or Flamingo) offer a potential solution by describing images freely. However, these models are often computationally expensive, verbose, and require complex prompting to produce simple object tags. This paper introduces NXTP (Next Token Prediction for recognition), a novel framework that treats object recognition as a next-token prediction task. By leveraging a truncated language decoder and a customized attention mechanism, NXTP enables efficient, open-vocabulary object labeling that is significantly faster than standard generative approaches.
Methodology
NXTP utilizes a pretrained CLIP image encoder and a language decoder based on the LLaMA architecture. The core innovation lies in how the model processes information to ensure both efficiency and accuracy.
Non-causal Attention Mask
Standard Large Language Models (LLMs) use a causal mask where each token can only attend to preceding tokens. NXTP modifies this structure to better suit the nature of image recognition. In this framework, image tokens act as a prefix that all subsequent tokens can see simultaneously. Crucially, the attention mask decouples tokens belonging to different object labels (e.g., "cat" and "sofa"). While all labels attend to the image tokens, they do not attend to each other. This reflects the conceptual independence of objects within a single scene.
Efficiency via Truncation and Parallelism
The authors discovered that the full depth of a 32-block LLM is unnecessary for the task of recognition. By using only the first 6 blocks of LLaMA-7B, the model retains nearly all its recognition performance while drastically reducing its parameter count and inference time. Furthermore, because labels are decoupled in the attention mask, NXTP can perform one-shot sampling. The model predicts the initial tokens of multiple potential labels in parallel and completes them simultaneously, rather than generating them one-by-one.
Experimental Setup
The models were trained at two scales: G3M (3 million image-text pairs) and G70M (70 million pairs). To create training data, the authors used NLTK to extract and lemmatize nouns from raw image captions, providing a weakly-supervised signal for the LLM to learn visual-linguistic mappings. Evaluation was conducted on the validation splits of CC3M, COCO, and OpenImages V7. Given the open-vocabulary nature of the model, accuracy was measured using BERTScore, which rewards semantically similar predictions (e.g., "automobile" for "car").
Key Results
NXTP demonstrates a significant leap in both accuracy and efficiency compared to existing benchmarks. On the COCO dataset, NXTP achieved an F1 score of 0.758, vastly outperforming CLIP (0.450) and even exceeding the performance of much larger models like InstructBLIP (0.725).
In terms of speed, the truncated 1.78B parameter version of NXTP is 4.5x faster than the full 7B version. When combined with the one-shot sampling technique, the system achieves an 18.1x speedup over standard greedy sampling on a full LLM. Qualitatively, the model exhibits "open-world" capabilities, identifying niche terms like "cloudscape" or "animation" that are typically absent from closed-set benchmarks.
Limitations and Future Work
Despite its strengths, NXTP is sensitive to the quality of training captions; noisy data can degrade performance. Additionally, while the model excels at general noun recognition, it currently struggles with highly fine-grained tasks, such as identifying specific species of birds. Future work may address the "competition issue" in one-shot sampling—where labels sharing the same starting token might conflict—and expand the framework to recognize complex noun phrases rather than single nouns.
Summary updated