TL;DR
This paper introduces CP-DETR, a novel universal object detection framework that leverages both text and visual prompts to improve zero-shot and fine-tuned detection performance. By introducing a hybrid prompt-visual encoder and innovative prompt generation methods, CP-DETR achieves state-of-the-art results across various benchmarks, demonstrating its ability to generalize to unseen categories and compete with specialized models.
What Problem Does This Paper Address?
Universal object detection aims to create a single model capable of detecting objects from any category, even those not seen during training. Current approaches often rely on textual descriptions (text prompts) to guide detection. However, these methods face challenges: (1) efficiently integrating prompt information with visual features and (2) mitigating alignment biases between text and image representations, especially in long-tailed or ambiguously described scenarios. This often leads to sub-optimal performance compared to closed-set detectors. The paper tackles these issues by proposing a more robust and versatile universal detection framework.
Methods
CP-DETR builds upon the DETR architecture and introduces several key innovations:
- Concept Prompt Generator: This module encodes various inputs (text descriptions, bounding boxes, annotations) into a unified "concept prompt" representation.
- Prompt Visual Hybrid Encoder: This is the core of CP-DETR's cross-modal interaction. It employs:
- Progressive Single-scale Fusion (PSF): Fuses prompt and image features scale-by-scale, starting from high-level semantic features, to avoid issues with semantic gaps between different feature levels.
- Multi-scale Fusion Gating (MFG): Enhances the fusion of critical information across scales.
- Auxiliary Supervision: To combat the sparsity of object queries in DETR and improve hybrid encoder training, CP-DETR adds:
- Auxiliary Detection Head: An anchor-based head provides denser supervision signals to image features.
- Prompt Multi-label Loss: Encourages concept prompts to learn from image information and reject irrelevant concepts.
- Concept Prompt Generation Methods:
- Text Prompt: Uses CLIP's text encoder for sentence-level representations.
- Visual Prompt: Encodes bounding box coordinates to extract visual examples, helping to avoid misalignments from incorrect descriptions.
- Optimized Prompt: Employs prompt tuning with super-class representation to align concept prompts with downstream data, improving performance without retraining the entire model.
Datasets
The paper utilizes a comprehensive set of datasets for training and evaluation:
- Training Datasets: Objects365 (O365), OpenImages (OI), V3Det, LVIS, COCO, GoldG, RefCOCO/+/g, Visual Genome (VG), and PhraseCut. Notably, the authors state they do not scale up data using pseudo-labeling, relying instead on publicly available datasets.
- Evaluation Benchmarks: COCO, LVIS, ODinW (including ODinW13 and ODinW35), and RefCOCO/+/g.
Key Insights and Ideas
- Visual Prompts as a Powerful Alternative: The paper highlights that visual prompts can significantly outperform text prompts in certain scenarios (e.g., interactive detection), suggesting that direct visual cues can be more effective at reducing alignment bias than potentially ambiguous text descriptions.
- Early Fusion with Hybrid Encoder is Crucial: The success of CP-DETR hinges on its hybrid encoder, which effectively fuses prompt and image features. The progressive, scale-by-scale approach is key to overcoming limitations of previous methods that fused all scales simultaneously.
- Auxiliary Supervision Boosts Generalization: The addition of auxiliary detection heads and prompt multi-label loss, while removed during inference, plays a vital role in facilitating cross-modal and cross-scale learning during training, leading to better overall generalization.
- Optimized Prompts for True Universality: The optimized prompt method, which fine-tunes prompts without altering the model weights, is presented as a way to truly reflect a detector's universality by aligning with downstream tasks.
Key Results
CP-DETR demonstrates strong performance across various settings:
- Zero-shot Detection: Achieved a new state-of-the-art of 32.2 AP on the ODinW35 benchmark with a Swin-L backbone. Also showed competitive results on COCO and LVIS, outperforming similarly sized models.
- Fine-tuning (Optimized Prompt): Achieved a record 73.1 AP on average for ODinW13, outperforming previous state-of-the-art models and demonstrating the ability to compete with fully fine-tuned specialist models.
- Interactive Detection (Visual Prompt): Achieved 68.4 AP on COCO val, significantly outperforming previous methods in interactive settings.
- Ablation Studies: Confirmed the effectiveness of the hybrid encoder (PSF and MFG), auxiliary supervision, and the super-class representation for optimized prompts.
Limitations and Future Work
- RefC Limitations: The paper notes that CP-DETR-T has limitations on the RefC benchmark, which they attribute to the pre-training data lacking descriptive sentences required for RefC evaluation.
- Dataset Quality Issues: The authors mention that some datasets used for evaluation (ODinW) had quality issues with annotated category names, which they addressed by following established evaluation setups.
- Future Work: While not explicitly detailed, the success of CP-DETR suggests future work could explore more sophisticated prompt generation techniques, further refine the hybrid encoder for even greater efficiency, and test the framework on a wider array of challenging downstream tasks and datasets. The paper's emphasis on a single model weight for all scenarios points towards a future of more generalized and adaptable AI systems.
Summary updated