LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction

Penghui Du, Yu Wang, Yifan Sun, Luting Wang, Yue Liao, Gang Zhang, Errui Ding, Yan Wang, Jingdong Wang, Si Liu

TL;DR

LaMI-DETR introduces a "Language Model Instruction" (LaMI) strategy to supercharge open-vocabulary object detection. By leveraging GPT-3.5 and T5 to generate rich visual descriptions and map inter-category relationships, the model overcomes the "shallow" understanding of traditional vision-language models, achieving a massive +7.8 point jump in rare object detection (APr) on the LVIS benchmark.

The Bottleneck: Shallow Concept Representation

Open-Vocabulary Object Detection (OVOD) aims to identify objects unseen during training. Most current methods rely on Vision-Language Models (VLMs) like CLIP. However, these models often suffer from "shallow" concept representation. CLIP’s text encoder frequently focuses on linguistic composition rather than visual semantics.

For instance, it might incorrectly associate "fireboat" and "fireweed" due to the shared prefix "fire," despite their vastly different appearances. Conversely, it may fail to recognize the visual similarity between a "sea lion" and a "dugong" because their names share no common tokens. This linguistic bias leads to poor generalization when the model encounters novel objects.

Comparison of concept representations. Visual descriptions help distinguish visually distinct but linguistically similar objects (left vs right) and group visually similar objects (bottom).

Furthermore, detectors often overfit to "base" categories seen during training, treating novel objects as background because the model has not learned to distinguish general foreground features from specific class labels.

Methodology: LaMI-DETR

The authors propose LaMI-DETR, a DETR-like detector that utilizes a frozen CLIP backbone and a novel instruction strategy. The architecture addresses the limitations of VLMs by injecting deeper semantic knowledge from Large Language Models (LLMs) like GPT-3.5 and T5.

The LaMI-DETR architecture, featuring visual concept sampling during training and LLM-guided inference for refined classification.

1. Inter-category Relationship Extraction

Instead of relying solely on category names, LaMI-DETR uses GPT-3.5 to generate detailed visual descriptions covering shape, color, and size. These descriptions are then embedded using T5 (Instructor Embedding) and clustered using K-means++. This process creates a "visual concept space" where categories are grouped by physical appearance rather than nomenclature.

2. Visual Concept Sampling

During training, the model utilizes these clusters to sample "easy negatives"—categories that are visually distinct from the ground truth. This strategy prevents the model from over-optimizing on base classes, encouraging it to learn generalized foreground features that translate better to unseen objects in an open-vocabulary setting.

3. Language Embedding Fusion and Inference

Textual knowledge is fused directly into the detector's object queries, helping the model "look" for specific semantic features. At inference time, if the model encounters visually similar categories (e.g., sea lion vs. dugong), it queries GPT-3.5 for discriminative descriptions. These descriptions highlight specific features that distinguish the two, which are then encoded by CLIP to refine the final classification score.

Key Results and Insights

LaMI-DETR was evaluated on the LVIS benchmark, which features 1,203 categories. The results demonstrate the power of LLM-guided instruction:

  • State-of-the-Art Performance: On OV-LVIS, LaMI-DETR achieved 43.4 APr (Average Precision for rare classes), a 7.8-point improvement over previous state-of-the-art methods.
  • Zero-Shot Generalization: In zero-shot settings (no fine-tuning on the target dataset), it reached 37.8 APr, outperforming the OWL-ST baseline by 2.9 points.
  • Efficiency: Despite the integration of LLMs, the model maintains a practical inference speed of 4.5 FPS on an NVIDIA V100, significantly faster than high-performance alternatives like GLIP-T (0.12 FPS).

Conclusion

LaMI-DETR demonstrates that LLMs like GPT-3.5 and T5 are superior "visual teachers" compared to standard VLMs. By clustering categories based on appearance and using discriminative descriptions, LaMI-DETR bridges the gap between linguistic labels and visual reality, setting a new standard for robust open-vocabulary object detection.

Summary updated

BibTeX
@inproceedings{du2024lami,
  author = {Penghui Du and Yu Wang and Yifan Sun and Luting Wang and Yue Liao and Gang Zhang and Errui Ding and Yan Wang and Jingdong Wang and Si Liu},
  title = {{LaMI-DETR: Open-Vocabulary Detection with Language Model Instruction}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73337-6\_18},
}
Figure