Abstract
DQ-DETR introduces a dual query framework to address Phrase Extraction and Grounding (PEG), a task requiring simultaneous phrase identification and object localization. By decoupling content queries while sharing positional information, the model achieves state-of-the-art results and introduces Cross-modal Average Precision (CMAP) for more robust evaluation.
Introduction: The PEG Challenge
Traditional visual grounding tasks like Referring Expression Comprehension (REC) and Phrase Grounding often operate in "clean" environments where phrases are pre-identified. In real-world scenarios, systems must perform Phrase Extraction and Grounding (PEG)—extracting noun phrases from raw text and locating corresponding objects in an image. This involves a 1D segmentation task and a 2D detection task, which often suffer from "modality interference" when using a single query for both visual and linguistic features.
Methodology: Dual Queries and CMAP
DQ-DETR treats PEG as a dual detection problem using a pair of queries: an Image Query and a Text Query. These queries share a positional component (looking at the same region in joint space) but maintain distinct content parts focused on visual and linguistic features, respectively. This decoupling prevents feature conflict and accelerates convergence.
To further refine the model's focus, Text Mask-Guided Attention uses predicted 1D text masks to filter transformer attention, ensuring the model attends only to relevant words for each object.
Cross-modal Average Precision (CMAP)
Standard metrics like Recall@1 are insufficient when a single phrase (e.g., "two pandas" or "two balls") refers to multiple distinct objects. The authors propose Cross-modal Average Precision (CMAP), which works like the mAP metric in object detection but requires a match in both the image (IoU) and the text (1D mask overlap).
Results and Insights
DQ-DETR establishes new state-of-the-art performance on benchmarks like Flickr30k Entities and RefCOCO. Notably, it outperformed MDETR on Flickr30k with half the training epochs (25 vs 50), achieving 76.0% CMAP50.
Key insights include:
- Decoupling is Essential: Separating content queries while sharing spatial anchors avoids feature alignment "tug-of-war."
- Phrase Extraction as Segmentation: Treating phrase identification as a 1D segmentation task allows the use of advanced architectures like Mask2Former.
- Addressing Ambiguity: CMAP provides a clearer picture of true multi-modal understanding by penalizing models that find the right object but the wrong phrase.
Conclusion and Future Work
While DQ-DETR shows significant improvements, scaling to a higher number of queries remains a challenge due to data imbalance in typical datasets. Future research will explore scaling these dual-query models for more complex scenes with numerous potential objects without losing precision.
Summary updated