RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension

Yabing Wang, Zhuotao Tian, Zheng Qin, Sanping Zhou, Le WangAAAI 2025 · DOI

TL;DR

This paper introduces RefDetector, a novel matching-based approach for referring expression comprehension (REC) that tackles the limitations of existing methods by improving referent recall and simplifying the vision-language interaction. RefDetector achieves state-of-the-art results on standard REC benchmarks, demonstrating its effectiveness and generalizability across different object detectors.

What Problem Does This Paper Address?

Referring Expression Comprehension (REC) is a crucial task in computer vision and natural language processing, aiming to locate a specific object in an image based on a descriptive text phrase. While object detection has made strides, it's limited by predefined categories. REC, on the other hand, deals with arbitrary descriptions. Existing REC methods primarily fall into two categories: regression-based and matching-based. Regression-based methods directly predict bounding box coordinates but can be computationally demanding and require extensive parameter tuning. Matching-based methods, while promising due to their ability to leverage pre-trained object detectors, have been under-explored and suffer from two main issues: the "mismatch problem" (where proposals are incorrectly filtered) and overly complex fusion mechanisms that limit adaptability. This paper aims to address these limitations by proposing a simpler, more effective, and generalizable matching-based framework.

Methods

RefDetector introduces several key innovations:

  • Heuristic Proposal Selection: Instead of relying on fixed confidence thresholds that can lead to the "mismatch problem," RefDetector employs a flexible "top-k" strategy for selecting proposals. This ensures that more potentially relevant proposals are retained, improving referent recall.
  • Transformer-based Vision-Language Interaction Module: To overcome the complexity of manually designed fusion mechanisms in prior work, RefDetector utilizes a straightforward transformer encoder-based module. This module effectively fuses visual and textual semantic information by leveraging cross-attention between proposal features and image features, and between proposal features and text features.
  • Expression-Proposal Alignment Objectives: Three objectives are introduced to enhance the model's ability to learn accurate matching:
  • Expression-Proposal Contrastive Learning (EPCL): Encourages higher scores for matched expression-proposal pairs than unmatched ones.
  • Expression-Proposal Matching (EPM): A matching head predicts matching logits using cross-entropy loss.
  • Hard Proposal Learning (HPL): Helps the model distinguish between matched and "hard negative" proposals.
  • DETR Integration: The paper explores integrating RefDetector with the modern DETR (DEtection TRansformer) object detector. DETR's learnable queries are found to provide rich contextual and positional information, leading to more distinctive proposal features.

Datasets

The paper evaluates RefDetector on three standard referring expression comprehension benchmarks:

  • RefCOCO: Contains approximately 50,000 referred objects across 19,994 images.
  • RefCOCO+: Similar to RefCOCO but with more complex expressions.
  • RefCOCOg: Features a larger number of images and expressions, with a focus on more challenging scenarios.

Key Insights and Ideas

  • Simplicity is Key: The paper demonstrates that a simpler, more general matching-based framework can outperform complex, hand-engineered solutions.
  • Top-k Proposals are Better: Moving away from strict confidence thresholds for proposal selection significantly improves referent recall and alleviates the mismatch problem.
  • DETR's Learnable Queries are Powerful: Integrating DETR's learnable queries into the proposal feature extraction process yields substantial performance gains due to their rich contextual and positional information.
  • Transformer Modules Generalize Well: The transformer-based interaction module is shown to be effective and adaptable, replacing intricate manual designs.
  • Matching-based Methods Still Have Untapped Potential: By revisiting and refining the matching-based paradigm, the authors show it can achieve state-of-the-art results.

Key Results

RefDetector achieves state-of-the-art performance on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks. For instance, when integrated with DETR, it surpasses the previous state-of-the-art method VG-LAW by significant margins across these datasets. The method also demonstrates remarkable efficiency, achieving strong results with only 18 training epochs on a single V100 GPU, outperforming methods requiring many more epochs and more powerful hardware. Ablation studies confirm the importance of each proposed component, particularly the visual and semantic interaction modules and the matching objectives.

Limitations and Future Work

The authors acknowledge that their work primarily focuses on the REC task. They suggest that future work could extend their approach to other related tasks such as open-vocabulary object detection and referring image segmentation. They also note that while their proposed interaction module is simple and effective, more sophisticated structures might yield further performance improvements, but this was not the primary focus of their study. The paper also highlights that their method is designed to work with frozen object detectors, meaning it doesn't aim to design a new detector itself.

Summary updated

BibTeX
@inproceedings{wang2025refdetector,
  author = {Yabing Wang and Zhuotao Tian and Zheng Qin and Sanping Zhou and Le Wang},
  title = {{RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i8.32866},
}

Loading related content...

Figure