DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding

Shilong Liu, Shijia Huang, Feng Li, Hao Zhang, Yaoyuan Liang, Hang Su, Jun Zhu, Lei Zhang

Abstract

DQ-DETR introduces a dual query framework to address Phrase Extraction and Grounding (PEG), a task requiring simultaneous phrase identification and object localization. By decoupling content queries while sharing positional information, the model achieves state-of-the-art results and introduces Cross-modal Average Precision (CMAP) for more robust evaluation.

Introduction: The PEG Challenge

Traditional visual grounding tasks like Referring Expression Comprehension (REC) and Phrase Grounding often operate in "clean" environments where phrases are pre-identified. In real-world scenarios, systems must perform Phrase Extraction and Grounding (PEG)—extracting noun phrases from raw text and locating corresponding objects in an image. This involves a 1D segmentation task and a 2D detection task, which often suffer from "modality interference" when using a single query for both visual and linguistic features.

Comparison of visual grounding tasks: DET, REC, Phrase Grounding, and the proposed PEG.

Methodology: Dual Queries and CMAP

DQ-DETR treats PEG as a dual detection problem using a pair of queries: an Image Query and a Text Query. These queries share a positional component (looking at the same region in joint space) but maintain distinct content parts focused on visual and linguistic features, respectively. This decoupling prevents feature conflict and accelerates convergence.

To further refine the model's focus, Text Mask-Guided Attention uses predicted 1D text masks to filter transformer attention, ensuring the model attends only to relevant words for each object.

Cross-modal Average Precision (CMAP)

Standard metrics like Recall@1 are insufficient when a single phrase (e.g., "two pandas" or "two balls") refers to multiple distinct objects. The authors propose Cross-modal Average Precision (CMAP), which works like the mAP metric in object detection but requires a match in both the image (IoU) and the text (1D mask overlap).

Comparison of evaluation metrics for multi-object phrases, highlighting the robustness of CMAP.

Results and Insights

DQ-DETR establishes new state-of-the-art performance on benchmarks like Flickr30k Entities and RefCOCO. Notably, it outperformed MDETR on Flickr30k with half the training epochs (25 vs 50), achieving 76.0% CMAP50.

Key insights include:

  • Decoupling is Essential: Separating content queries while sharing spatial anchors avoids feature alignment "tug-of-war."
  • Phrase Extraction as Segmentation: Treating phrase identification as a 1D segmentation task allows the use of advanced architectures like Mask2Former.
  • Addressing Ambiguity: CMAP provides a clearer picture of true multi-modal understanding by penalizing models that find the right object but the wrong phrase.

Conclusion and Future Work

While DQ-DETR shows significant improvements, scaling to a higher number of queries remains a challenge due to data imbalance in typical datasets. Future research will explore scaling these dual-query models for more complex scenes with numerous potential objects without losing precision.

Summary updated

Abstract

In this paper, we study the problem of visual grounding by considering both phrase extraction and grounding (PEG). In contrast to the previous phrase-known-at-test setting, PEG requires a model to extract phrases from text and locate objects from image simultaneously, which is a more practical setting in real applications. As phrase extraction can be regarded as a 1D text segmentation problem, we formulate PEG as a dual detection problem and propose a novel DQ-DETR model, which introduces dual queries to probe different features from image and text for object prediction and phrase mask prediction. Each pair of dual queries are designed to have shared positional parts but different content parts. Such a design effectively alleviates the difficulty of modality alignment between image and text (in contrast to a single query design) and empowers Transformer decoder to leverage phrase mask-guided attention to improve the performance. To evaluate the performance of PEG, we also propose a new metric CMAP (cross-modal average precision), analogous to the AP metric in object detection. The new metric overcomes the ambiguity of Recall@1 in many-box-to-one-phrase cases in phrase grounding. As a result, our PEG pre-trained DQ-DETR establishes new state-of-the-art results on all visual grounding benchmarks with a ResNet-101 backbone. For example, it achieves 91.04% and 83.51% in terms of recall rate on RefCOCO testA and testB with a ResNet-101 backbone.
BibTeX
@inproceedings{liu2023dq,
  author = {Shilong Liu and Shijia Huang and Feng Li and Hao Zhang and Yaoyuan Liang and Hang Su and Jun Zhu and Lei Zhang},
  title = {{DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25261},
}
Figure