DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding

Shilong Liu, Yaoyuan Liang, Feng Li, Shijia Huang, Hao Zhang, Hang Su, Jun Zhu, Lei Zhang

Introduction: The Challenge of Phrase Extraction and Grounding

In traditional visual grounding, models are typically provided with a specific phrase, such as "the blue car," and tasked with locating it within an image. However, real-world scenarios often involve raw, un-annotated captions. This necessitates a more practical task known as Phrase Extraction and Grounding (PEG). In PEG, a model must automatically extract noun phrases from a raw caption and simultaneously link them to their corresponding regions in the image.

Comparison of vision-language tasks, highlighting Phrase Extraction and Grounding (PEG).

Existing state-of-the-art models, such as MDETR, utilize a single "query" vector to predict both the bounding box and the corresponding text phrase. This approach often suffers from a "modality gap," where forcing a single vector to align complex visual and linguistic features leads to slow training and suboptimal performance. Furthermore, standard metrics like Recall@1 can be ambiguous when a single phrase refers to multiple objects (e.g., "two balls").

Methodology: The Dual Query Architecture

To address these limitations, the authors introduce DQ-DETR (Dual Query Detection Transformer). The core innovation lies in its dual-query mechanism, which decouples the content representation for different modalities while maintaining spatial alignment.

Architectural comparison between standard detection, single-query PEG, and the proposed DQ-DETR dual-query approach.

Key Components

  • Dual Query Design: Instead of a single query per object, DQ-DETR employs a pair of queries: an Image Query for visual features (bounding box regression) and a Text Query for linguistic features (predicting a 1D text mask to identify phrases).
  • Decoupled Content, Shared Position: These queries share the same positional information ("where") but have distinct content parts ("what"). This ensures that the model understands both queries refer to the same entity despite looking at different data types.
  • Text Mask-Guided Attention: Inspired by 2D segmentation techniques, the model uses the predicted 1D text mask to guide the Transformer’s attention, focusing specifically on relevant words for a given object.
  • CMAP Metric: The authors propose Cross-modal Average Precision (CMAP) to evaluate box-phrase pairs more robustly than traditional recall metrics, similar to the mAP used in standard object detection.

Evaluation and Results

DQ-DETR was evaluated on major benchmarks including Flickr30k Entities, RefCOCO/+/g, and COCO. By decoupling the modalities, the model achieves significantly faster convergence and superior accuracy.

On the Flickr30k Entities benchmark, DQ-DETR achieved 76.0% CMAP50 and 83.2% Recall@1, outperforming MDETR by 5.8 points in CMAP. In Zero-Shot Detection on COCO, it outperformed MDETR by 4.7 AP, demonstrating that improved multi-modal alignment benefits pure visual tasks as well.

Conclusion

DQ-DETR demonstrates that modality alignment is a significant bottleneck in single-query architectures. By treating phrase extraction as a 1D segmentation problem and utilizing a dual-query framework, the model achieves state-of-the-art performance and efficiency. While it still trails specialized "vision-only" models in pure detection, it represents a major step forward for integrated vision-language understanding. Future work may involve scaling with larger backbones and handling more complex linguistic structures.

Summary updated

BibTeX
@inproceedings{liu2023dq,
  author = {Shilong Liu and Yaoyuan Liang and Feng Li and Shijia Huang and Hao Zhang and Hang Su and Jun Zhu and Lei Zhang},
  title = {{DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25261},
}
Figure