Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point

Peizhi Zhao, Shiyi Zheng, Wenye Zhao, Dongsheng Xu, Pijian Li, Yi Cai, Qingbao HuangAAAI 2024 · DOI

TL;DR

This paper introduces a novel point-based two-stage framework for Referring Expression Comprehension (REC) that reframes the task as a binary classification problem. By using "soft ground-truth" metrics like center-ness and mass-ness, the method effectively bridges the gap between comprehension and localization, leading to improved performance and a unified approach for both grounding and segmentation tasks.

What Problem Does This Paper Address?

Referring Expression Comprehension (REC) is a crucial task in vision and language, aiming to locate a specific object in an image based on a textual description. Existing methods fall into two main categories: one-stage and two-stage. One-stage methods, while leveraging rich context, are computationally expensive and data-hungry. Two-stage methods decouple the problem into region proposal and then matching, but suffer from fragmented spatial context due to sparse region proposals and a mismatch between training and inference data. This paper aims to create a more efficient and effective two-stage system that overcomes these limitations.

Methods

The core innovation of this paper is a point-based two-stage framework. Instead of relying on traditional region proposals, it reformulates REC into:

  1. Point-based Cross-Modal Comprehension: This stage aligns visual and linguistic features. Crucially, it introduces "soft ground-truth" metrics:
  2. Center-ness: Measures how close a point is to the center of the target bounding box.
  3. Mass-ness: Measures the foreground enrichment of a point based on segmentation masks. These metrics are derived from the ground-truth bounding boxes and segmentation masks, allowing the model to learn from dense point-level correlations rather than sparse regions. This stage is optimized using a binary classification loss based on these soft ground-truth values.

  4. Point-based Instance Localization: This stage leverages a vision localization module (like FCOS for grounding or SOLO for segmentation) that is compatible with point-based predictions.

A key advantage is that the same comprehension stage, optimized with both center-ness and mass-ness, can naturally support both grounding and segmentation tasks, enabling multi-task learning without additional heads. Finally, an IoU-based Non-Maximum Suppression (NMS) is proposed as a post-processing step to refine the final predictions from the localization stage.

Datasets

The paper evaluates its method on three standard datasets for Referring Expression Comprehension:

  • RefCOCO
  • RefCOCO+
  • RefCOCOg

These datasets are collected from MSCOCO and vary in complexity, with RefCOCOg featuring longer and more realistic expressions.

Key Insights and Ideas

  • Shifting from Regions to Points: The most significant insight is the move from region-based processing to point-based processing. This allows for a more holistic understanding of spatial context and avoids the limitations of sparse region proposals.
  • Soft Ground-Truth for Bridging Gaps: The introduction of center-ness and mass-ness as soft ground-truth metrics is a clever way to approximate the complex REC task as a more manageable binary classification problem. This directly addresses the "gap" between training and inference data that plagues traditional two-stage methods.
  • Unified Framework for Grounding and Segmentation: By using point-level metrics that are consistent across both grounding (center-ness) and segmentation (mass-ness), the framework naturally supports multi-task learning. This simplifies the model architecture and training process.
  • Efficient Training: The reformulation as a classification problem significantly reduces the computational burden and training epochs required compared to transformer-based one-stage methods.

Key Results

The proposed point-based two-stage framework, PBREC-MT, achieves state-of-the-art results on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks for both Referring Expression Grounding (REG) and Referring Expression Segmentation (RES).

  • REG: PBREC-MT shows significant improvements over conventional two-stage methods, outperforming Ref-NMS by 4.77%, 6.60%, and 3.51% on the three datasets. It also achieves better performance than transformer-based one-stage methods like TransVG with substantially fewer training epochs.
  • RES: The method demonstrates substantial gains over two-stage methods, with absolute advantages of 9.98%/7.66%/12.70% on RefCOCO, RefCOCO+, and RefCOCOg, respectively.
  • Qualitative Results: Visualizations show that the model effectively focuses on the referred instance using attention maps and that the center-ness and mass-ness predictions align well with the target. The IoU-based NMS is shown to refine predictions and correct errors caused by occlusion or incomplete cross-modal comprehension.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the recall of region proposals from detectors like FCOS is high (around 97%), the performance ceiling for traditional two-stage methods on specific challenging splits (like RefCOCO testB) is limited by the inherent nature of anchor-based detectors. The proposed method aims to mitigate this but doesn't entirely eliminate the reliance on an underlying detector.
  • Future Work: The authors express interest in extending their point-based two-stage paradigm to more challenging scenarios, such as open-domain REC and zero-shot learning.

Summary updated

BibTeX
@inproceedings{zhao2024rethinking,
  author = {Peizhi Zhao and Shiyi Zheng and Wenye Zhao and Dongsheng Xu and Pijian Li and Yi Cai and Qingbao Huang},
  title = {{Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28580},
}
Figure