DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

Siheng Wang, Yanshu Li, Bohan Hu, Zhengdao Li, Haibo Zhan, Linshan Li, Weiming Liu, Ruizhi Qian, Guangxin Wu, Hao Zhang, Jifeng Shen, Piotr Koniusz, Zhengtao Yao, Junhao Dong, Qiang Sun

TL;DR

DeCo-DETR is a vision-centric framework that makes open-vocabulary object detection (OVOD) both faster and more accurate by decoupling semantic reasoning from spatial localization. By replacing heavy inference-time text encoders with a hierarchical semantic prototype pool, it achieves state-of-the-art zero-shot performance while running up to twice as fast as traditional fusion-based models.

What Problem Does This Paper Address?

Open-vocabulary object detection (OVOD) is the "holy grail" of computer vision: a system that can detect and recognize objects it has never seen during training, simply by being given a text description. While Large Vision-Language Models (LVLMs) like CLIP have made this possible, two major hurdles remain:

  1. Computational Bottlenecks: Most current models rely on heavy text encoders (like BERT or CLIP-Text) at inference time to process category names. This makes them too slow for real-time applications like autonomous driving.
  2. Optimization Conflicts: There is an inherent tension between learning where an object is (localization) and what it is in a broad, open-world sense (semantic alignment). In many models, training for precise bounding boxes actually degrades the model's ability to generalize to new, unseen categories.

Methods

The authors introduce DeCo-DETR (Decoupled Cognition DETR), which uses three novel components to solve these issues:

  • Dynamic Hierarchical Concept Pool (DHCP): Instead of running a text encoder every time you want to detect something, DeCo-DETR builds a "library" of semantic prototypes. They use LLaVA to generate detailed descriptions of image regions and CLIP to align them. These are then clustered into a hierarchy: Coarse prototypes (broad categories) and Fine prototypes (specific attributes and variations).
  • Hierarchical Knowledge Distillation (Hi-Know DPA): This module aligns the standard DETR object queries with the prototype pool. It allows the detector to "borrow" rich semantic knowledge from the pre-trained VLM without needing the VLM present during inference.
  • Parametric Decoupling Training (PD-DuGi): This is the core architectural shift. The authors separate the localization stream (bounding boxes) from the cognition stream (classification). Crucially, they use Gradient Isolation: gradients from the semantic alignment task are not allowed to interfere with the features used for localization.
Figure 1: Three-stage pipeline of DeCo-DETR. (a) DHCP constructs a hierarchical prototype mem- ory from region-level descriptions via LLaVA generation and CLIP-based filtering, capturing both coarse and fine-grained semantics. (b) Hi-Know DPA projects detector queries into the shared em- bedding space and enhances them through prototype aggregation for efficient open-set knowledge transfer. (c) PD-DuGi decouples localization and semantic alignment into two optimization streams, mitigating task interference. At inference, the prototype pool provides semantic priors, and the de- coupled decoder jointly predicts bounding boxes and category semantics without a text encoder.

Datasets

The model was evaluated on two primary OVOD benchmarks:

  • OV-COCO: A variant of the MSCOCO dataset with 48 "base" classes for training and 17 "novel" classes for zero-shot testing.
  • OV-LVIS: A more challenging, long-tailed dataset with 866 base categories and 337 rare/novel categories.

Key Insights and Ideas

  • Vision-Centric Inference is Key: By moving the "textual heavy lifting" to an offline prototype pool, the model becomes purely vision-based at inference time. This eliminates the latency of text encoders entirely.
  • Hierarchy Captures Nuance: The authors found that using only category-level prototypes wasn't enough. Adding fine-grained prototypes (via DBSCAN clustering) improved novel class detection by a massive 10.5 points, proving that "details matter" for generalization.
  • Gradient Interference is Real: The paper demonstrates that the "trade-off" between seen and unseen class accuracy is largely due to gradient conflict. By isolating these optimization paths, DeCo-DETR improves both localization precision and open-world recognition simultaneously.

Key Results

DeCo-DETR sets new benchmarks for efficiency and accuracy:

  • Speed: It achieves a latency of 135ms (7.4 FPS) on an NVIDIA RTX 4090. Compared to fusion-based models like Grounding DINO (~280ms), it is roughly 2x faster.
  • Accuracy on OV-COCO: It reached 41.3% $AP_{50}^{novel}$, surpassing the previous state-of-the-art (LBP) by 3.5 points.
  • Accuracy on OV-LVIS: On the challenging rare classes, it achieved 29.4% $AP_r$, establishing a new record for the dataset.
Table 1: OV-COCO comparison (AP50) across a wide range of widely used OVOD methods.
Table 2: OV-LVIS comparison (AP) across a wide range of widely used OVOD methods.

Limitations and Future Work

Despite its success, the authors note a few areas for improvement:

  • VLM Dependency: The initial construction of the concept pool still relies on heavy models like LLaVA and CLIP, which requires significant offline compute.
  • Extreme Long-Tail: While it performs well on LVIS, the model still struggles with extremely similar fine-grained categories in very sparse data distributions.
  • Static Images: The current framework is designed for static images and hasn't yet been extended to handle the temporal dynamics of video sequences.

Summary updated

Abstract

Open-vocabulary object detection (OVOD) enables models to recognize objects beyond predefined categories, but existing approaches remain limited in practical deployment. On the one hand, multimodal designs often incur substantial computational overhead due to their reliance on text encoders at inference time. On the other hand, tightly coupled training objectives introduce a trade-off between closed-set detection accuracy and open-world generalization. Thus, we propose Decoupled Cognition DETR (DeCo-DETR), a vision-centric framework that addresses these challenges through a unified decoupling paradigm. Instead of depending on online text encoding, DeCo-DETR constructs a hierarchical semantic prototype space from region-level descriptions generated by pre-trained LVLMs and aligned via CLIP, enabling efficient and reusable semantic representation. Building upon this representation, the framework further disentangles semantic reasoning from localization through a decoupled training strategy, which separates alignment and detection into parallel optimization streams. Extensive experiments on standard OVOD benchmarks demonstrate that DeCo-DETR achieves competitive zero-shot detection performance while significantly improving inference efficiency. These results highlight the effectiveness of decoupling semantic cognition from detection, offering a practical direction for scalable OVOD systems.
BibTeX
@inproceedings{wang2026deco,
  author = {Siheng Wang and Yanshu Li and Bohan Hu and Zhengdao Li and Haibo Zhan and Linshan Li and Weiming Liu and Ruizhi Qian and Guangxin Wu and Hao Zhang and Jifeng Shen and Piotr Koniusz and Zhengtao Yao and Junhao Dong and Qiang Sun},
  title = {{DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=3UDlRUf1es},
}
Figure