Abstract
Referring Expression Segmentation (RES) has attracted rising attention, aiming to identify and segment objects based on natural language expressions. While substantial progress has been made in RES, the emergence of Generalized Referring Expression Segmentation (GRES) introduces new challenges by allowing expressions to describe multiple objects or lack specific object references. Existing RES methods, usually rely on sophisticated encoder-decoder and feature fusion modules, and are difficult to generate class prototypes that match each instance individually when confronted with the complex referent and binary labels of GRES. In this paper, reevaluating the differences between RES and GRES, we propose a novel Model with Adaptive Binding Prototypes (MABP) that adaptively binds queries to object features in the corresponding region. It enables different query vectors to match instances of different categories or different parts of the same instance, significantly expanding the decoder's flexibility, dispersing global pressure across all queries, and easing the demands on the encoder. Experimental results demonstrate that MABP significantly outperforms state-of-the-art methods in all three splits on gRefCOCO dataset. Meanwhile, MABP also surpasses state-of-the-art methods on RefCOCO+ and G-Ref datasets, and achieves very competitive results on RefCOCO.
Introduction
Traditional Referring Expression Segmentation (RES) is built on the assumption that a single natural language expression describes exactly one object. However, real-world applications often involve more complex scenarios where expressions may refer to multiple targets (e.g., "the two dogs and the man") or no targets at all (e.g., referring to a "cat" in a scene where none exists). This broader task is known as Generalized Referring Expression Segmentation (GRES).
Existing RES models typically employ a "greedy" approach, attempting to summarize all foreground objects into a single global class prototype. This strategy fails in GRES due to three primary issues: Category Conflict, where a single prototype cannot represent multiple distinct classes; No-Target Instability, where attention mechanisms hallucinate targets in empty scenes; and Encoder Pressure, which forces the backbone to map diverse objects into a unified feature space.
Methodology: Model with Adaptive Binding Prototypes (MABP)
To address these challenges, we propose MABP, a framework that shifts from global prototypes to Adaptive Binding Prototypes. The architecture is defined by three key innovations:
- Region-Text Specific Query Generator: Instead of random initialization, MABP utilizes learnable region embeddings. These are fused with linguistic features via cross-attention, creating queries that are spatially and semantically aware of their designated image regions.
- Mixed Modal Decoder (MMD): To stabilize performance on "no-target" samples, the decoder explicitly incorporates linguistic features as anchors during the self-attention phase. This ensures that the attention map remains manageable even when no visual target is present, leading to reliable "empty" predictions.
- Regional Supervision Head (RSH): This core mechanism partitions the feature map into a grid. Each query is strictly supervised by the ground truth of its corresponding patch, forcing queries to specialize in local details and dispersing the global pressure across the entire query set.
Experimental Evaluation
MABP was evaluated on several benchmarks, including gRefCOCO (the primary GRES dataset), standard RES datasets (RefCOCO, RefCOCO+, G-Ref), and the MeViS video benchmark.
Key Results
- State-of-the-Art Performance: On gRefCOCO, MABP outperformed the previous leading model (ReLA) by approximately 2% in cIoU and 3% in gIoU across all splits.
- No-Target Reliability: MABP showed a significant jump in no-target accuracy (N-acc), improving from 57.51% to 64.55%.
- Fine-Grained Precision: The model demonstrated notable improvements in [email protected], indicating superior capability in capturing fine details and small objects compared to previous methods.
- Generalization: Despite its GRES focus, MABP achieved competitive or superior results on classic RES datasets like RefCOCO+ and G-Ref.
Conclusion and Future Work
MABP demonstrates that a "divide and conquer" approach—binding queries to specific spatial regions—allows for more flexible and stable segmentation in complex linguistic scenarios. By preserving the encoder's pre-trained knowledge and using language as a stabilizing anchor, MABP sets a new standard for GRES.
Future research will explore deformable partitioning to move beyond fixed grid boundaries and improve the handling of "near-miss" no-target cases, where the scene contains objects very similar to the described target.
Summary updated