Object Recognition as Next Token Prediction

Kaiyu Yue, Bor-Chun Chen, Jonas Geiping, Hengduo Li, Tom Goldstein, Ser-Nam Lim

Introduction

Traditional object recognition has long been divided into two distinct paradigms. Closed-set classification models, such as ResNet, rely on fixed linear layers to predict a predefined set of categories, making them incapable of recognizing concepts outside their training set. Conversely, retrieval-based models like CLIP match image embeddings against a gallery of text descriptions. While more flexible, retrieval-based methods are constrained by the quality and breadth of the predefined gallery; if a concept is missing from the list, the model remains "blind" to it.

Generative multimodal models (e.g., LLaVA or Flamingo) offer a potential solution by describing images freely. However, these models are often computationally expensive, verbose, and require complex prompting to produce simple object tags. This paper introduces NXTP (Next Token Prediction for recognition), a novel framework that treats object recognition as a next-token prediction task. By leveraging a truncated language decoder and a customized attention mechanism, NXTP enables efficient, open-vocabulary object labeling that is significantly faster than standard generative approaches.

Methodology

NXTP utilizes a pretrained CLIP image encoder and a language decoder based on the LLaMA architecture. The core innovation lies in how the model processes information to ensure both efficiency and accuracy.

Non-causal Attention Mask

Standard Large Language Models (LLMs) use a causal mask where each token can only attend to preceding tokens. NXTP modifies this structure to better suit the nature of image recognition. In this framework, image tokens act as a prefix that all subsequent tokens can see simultaneously. Crucially, the attention mask decouples tokens belonging to different object labels (e.g., "cat" and "sofa"). While all labels attend to the image tokens, they do not attend to each other. This reflects the conceptual independence of objects within a single scene.

The NXTP non-causal attention mask (M) decouples labels while treating image tokens (Xv) as a shared prefix.

Efficiency via Truncation and Parallelism

The authors discovered that the full depth of a 32-block LLM is unnecessary for the task of recognition. By using only the first 6 blocks of LLaMA-7B, the model retains nearly all its recognition performance while drastically reducing its parameter count and inference time. Furthermore, because labels are decoupled in the attention mask, NXTP can perform one-shot sampling. The model predicts the initial tokens of multiple potential labels in parallel and completes them simultaneously, rather than generating them one-by-one.

Experimental Setup

The models were trained at two scales: G3M (3 million image-text pairs) and G70M (70 million pairs). To create training data, the authors used NLTK to extract and lemmatize nouns from raw image captions, providing a weakly-supervised signal for the LLM to learn visual-linguistic mappings. Evaluation was conducted on the validation splits of CC3M, COCO, and OpenImages V7. Given the open-vocabulary nature of the model, accuracy was measured using BERTScore, which rewards semantically similar predictions (e.g., "automobile" for "car").

Key Results

NXTP demonstrates a significant leap in both accuracy and efficiency compared to existing benchmarks. On the COCO dataset, NXTP achieved an F1 score of 0.758, vastly outperforming CLIP (0.450) and even exceeding the performance of much larger models like InstructBLIP (0.725).

Precision-recall curves across different benchmarks show NXTP (Ours) consistently outperforming CLIP and Flamingo-MPT.

In terms of speed, the truncated 1.78B parameter version of NXTP is 4.5x faster than the full 7B version. When combined with the one-shot sampling technique, the system achieves an 18.1x speedup over standard greedy sampling on a full LLM. Qualitatively, the model exhibits "open-world" capabilities, identifying niche terms like "cloudscape" or "animation" that are typically absent from closed-set benchmarks.

Limitations and Future Work

Despite its strengths, NXTP is sensitive to the quality of training captions; noisy data can degrade performance. Additionally, while the model excels at general noun recognition, it currently struggles with highly fine-grained tasks, such as identifying specific species of birds. Future work may address the "competition issue" in one-shot sampling—where labels sharing the same starting token might conflict—and expand the framework to recognize complex noun phrases rather than single nouns.

Summary updated

Abstract

We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels. To ground this prediction process in auto-regression, we customize a non-causal attention mask for the decoder, incorporating two key features: modeling tokens from different labels to be independent, and treating image tokens as a prefix. This masking mechanism inspires an efficient method - one-shot sampling - to simultaneously sample tokens of multiple labels in parallel and rank generated labels by their probabilities during inference. To further enhance the efficiency, we propose a simple strategy to construct a compact decoder by simply discarding the intermediate blocks of a pretrained language model. This approach yields a decoder that matches the full model's performance while being notably more efficient. The code is available at github.com/kaiyuyue/nxtp.
BibTeX
@inproceedings{yue2023object,
  author = {Kaiyu Yue and Bor-Chun Chen and Jonas Geiping and Hengduo Li and Tom Goldstein and Ser-Nam Lim},
  title = {{Object Recognition as Next Token Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure