TL;DR
This paper introduces a novel point-based two-stage framework for Referring Expression Comprehension (REC) that reframes the task as a binary classification problem. By using "soft ground-truth" metrics like center-ness and mass-ness, the method effectively bridges the gap between comprehension and localization, leading to improved performance and a unified approach for both grounding and segmentation tasks.
What Problem Does This Paper Address?
Referring Expression Comprehension (REC) is a crucial task in vision and language, aiming to locate a specific object in an image based on a textual description. Existing methods fall into two main categories: one-stage and two-stage. One-stage methods, while leveraging rich context, are computationally expensive and data-hungry. Two-stage methods decouple the problem into region proposal and then matching, but suffer from fragmented spatial context due to sparse region proposals and a mismatch between training and inference data. This paper aims to create a more efficient and effective two-stage system that overcomes these limitations.
Methods
The core innovation of this paper is a point-based two-stage framework. Instead of relying on traditional region proposals, it reformulates REC into:
- Point-based Cross-Modal Comprehension: This stage aligns visual and linguistic features. Crucially, it introduces "soft ground-truth" metrics:
- Center-ness: Measures how close a point is to the center of the target bounding box.
-
Mass-ness: Measures the foreground enrichment of a point based on segmentation masks. These metrics are derived from the ground-truth bounding boxes and segmentation masks, allowing the model to learn from dense point-level correlations rather than sparse regions. This stage is optimized using a binary classification loss based on these soft ground-truth values.
-
Point-based Instance Localization: This stage leverages a vision localization module (like FCOS for grounding or SOLO for segmentation) that is compatible with point-based predictions.
A key advantage is that the same comprehension stage, optimized with both center-ness and mass-ness, can naturally support both grounding and segmentation tasks, enabling multi-task learning without additional heads. Finally, an IoU-based Non-Maximum Suppression (NMS) is proposed as a post-processing step to refine the final predictions from the localization stage.
Datasets
The paper evaluates its method on three standard datasets for Referring Expression Comprehension:
- RefCOCO
- RefCOCO+
- RefCOCOg
These datasets are collected from MSCOCO and vary in complexity, with RefCOCOg featuring longer and more realistic expressions.
Key Insights and Ideas
- Shifting from Regions to Points: The most significant insight is the move from region-based processing to point-based processing. This allows for a more holistic understanding of spatial context and avoids the limitations of sparse region proposals.
- Soft Ground-Truth for Bridging Gaps: The introduction of center-ness and mass-ness as soft ground-truth metrics is a clever way to approximate the complex REC task as a more manageable binary classification problem. This directly addresses the "gap" between training and inference data that plagues traditional two-stage methods.
- Unified Framework for Grounding and Segmentation: By using point-level metrics that are consistent across both grounding (center-ness) and segmentation (mass-ness), the framework naturally supports multi-task learning. This simplifies the model architecture and training process.
- Efficient Training: The reformulation as a classification problem significantly reduces the computational burden and training epochs required compared to transformer-based one-stage methods.
Key Results
The proposed point-based two-stage framework, PBREC-MT, achieves state-of-the-art results on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks for both Referring Expression Grounding (REG) and Referring Expression Segmentation (RES).
- REG: PBREC-MT shows significant improvements over conventional two-stage methods, outperforming Ref-NMS by 4.77%, 6.60%, and 3.51% on the three datasets. It also achieves better performance than transformer-based one-stage methods like TransVG with substantially fewer training epochs.
- RES: The method demonstrates substantial gains over two-stage methods, with absolute advantages of 9.98%/7.66%/12.70% on RefCOCO, RefCOCO+, and RefCOCOg, respectively.
- Qualitative Results: Visualizations show that the model effectively focuses on the referred instance using attention maps and that the center-ness and mass-ness predictions align well with the target. The IoU-based NMS is shown to refine predictions and correct errors caused by occlusion or incomplete cross-modal comprehension.
Limitations and Future Work
- Limitations: The paper acknowledges that while the recall of region proposals from detectors like FCOS is high (around 97%), the performance ceiling for traditional two-stage methods on specific challenging splits (like RefCOCO testB) is limited by the inherent nature of anchor-based detectors. The proposed method aims to mitigate this but doesn't entirely eliminate the reliance on an underlying detector.
- Future Work: The authors express interest in extending their point-based two-stage paradigm to more challenging scenarios, such as open-domain REC and zero-shot learning.
Summary updated