Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation

Weize Li, Zhicheng Zhao, Haochen Bai, Fei Su

TL;DR

This paper introduces MABP (Model with Adaptive Binding Prototypes), a novel framework designed for Generalized Referring Expression Segmentation (GRES). By partitioning images into spatial regions and "binding" specific language-aware queries to those regions, the model effectively handles complex natural language prompts that describe multiple objects, different categories, or even no objects at all.

What Problem Does This Paper Address?

Traditional Referring Expression Segmentation (RES) operates on a simple rule: one sentence describes exactly one object. However, real-world scenarios are rarely that tidy. Generalized RES (GRES) was introduced to handle expressions that refer to multiple targets (e.g., "the two dogs and the man") or no targets at all (e.g., "the cat" when no cat is present).

Existing RES models often use a "greedy" approach, where a set of global query vectors tries to summarize all foreground objects into a single class prototype. This fails in GRES because:

  1. Category Conflict: A single prototype struggles to represent multiple distinct categories (like a "dog" and a "guy") simultaneously.
  2. No-Target Instability: When no target exists, the attention mechanisms in these models become unstable, often leading to "hallucinated" segmentations.
  3. Encoder Pressure: Forcing the encoder to map diverse objects into a single feature representation places an immense burden on the backbone network.

Methods

The authors propose MABP, which shifts from global prototypes to Adaptive Binding Prototypes. The architecture consists of three main innovations:

  • Region-Text Specific Query Generator: Instead of random initialization, the model uses learnable region embeddings. These are combined with linguistic features via cross-attention to create initial queries that already "know" which part of the image they are responsible for and what they are looking for.
  • Mixed Modal Decoder (MMD): To handle "no-target" samples, the decoder doesn't just look at visual features. It explicitly incorporates linguistic features as "placeholders" during the self-attention phase. This ensures that even if the image contains no targets, the attention map remains stable and manageable, leading to a clean "empty" prediction.
  • Regional Supervision Head (RSH): This is the core "binding" mechanism. The model partitions the feature map into a grid of patches. Each query is strictly supervised by the ground truth of its corresponding patch. This forces queries to specialize in local details and prevents the "global pressure" found in previous models.

Datasets

The model was evaluated on several key benchmarks:

  • gRefCOCO: The primary large-scale dataset for GRES, containing multi-target and no-target expressions.
  • RefCOCO / RefCOCO+ / G-Ref: Standard RES datasets used to test the model's generalization to classic single-target tasks.
  • MeViS: A large-scale benchmark for referring video object segmentation, used to test the upper limits of the model's flexibility.

Key Insights and Ideas

  • Divide and Conquer: By forcing queries to bind to specific spatial regions, the model allows different queries to represent different categories (e.g., one query handles the "dog" in the bottom-left while another handles the "man" in the center).
  • Stability through Language: Inverting the typical attention flow by using language features as anchors in the decoder significantly improves performance on "empty" samples. It prevents the model from "searching" too hard for a target that isn't there.
  • Preserving Pre-trained Knowledge: Standard GRES training often forces encoders to map different categories into the same "foreground" feature space. MABP’s regional approach allows the encoder to maintain its pre-trained ability to distinguish between classes, as the "merging" happens later in the regional heads.

Key Results

MABP achieved state-of-the-art (SOTA) results across multiple benchmarks:

  • gRefCOCO: Outperformed the previous best model (ReLA) by approximately 2% in cIoU and 3% in gIoU across all splits.
  • No-Target Performance: On the difficult "no-target" samples, MABP improved accuracy (N-acc) from 57.51% to 64.55%, a significant jump in reliability.
  • Precision at High Thresholds: The model showed a notable improvement in [email protected] (precision at a 0.9 IoU threshold), indicating it is much better at capturing fine details and small objects than its predecessors.
  • Generalization: Despite being designed for GRES, MABP also outperformed SOTA RES models on RefCOCO+ and G-Ref.

Limitations and Future Work

The authors identify two main areas for improvement:

  1. Rigid Partitioning: Currently, the model uses a "hard-split" strategy with fixed, coarse-grained grid boundaries. This can cause issues if a small object sits exactly on the boundary between two regions.
  2. Complex No-Target Scenarios: While MABP handles "obvious" no-target cases well (e.g., "laptop" when there is no laptop), it still struggles with "near-miss" cases where the expression is very similar to something in the image (e.g., "man in blue shirt" when there is a man in a purple shirt).

Future work may involve deformable partitioning (allowing regions to change shape) or adaptive merging of prototypes to handle these edge cases more gracefully.

Summary updated

Abstract

Referring Expression Segmentation (RES), which aims to identify and segment objects based on natural language expressions is garnering increased research attention. While substantial progress has been made in RES, the emergence of Generalized Referring Expression Segmentation (GRES) introduces new challenges by allowing the expressions to describe multiple objects or lack specific object references. Existing RES methods usually rely on sophisticated encoder-decoder and feature fusion modules, and have difficulty generating class prototypes that match each instance individually when confronted with the complex referent and binary labels of GRES. In this paper, reevaluating the differences between RES and GRES, we propose a novel Model with Adaptive Binding Prototypes (MABP) that adaptively binds queries to object features in the corresponding region. It enables different query vectors to match instances of different categories, or different parts of the same instance, significantly expanding the decoder's flexibility, dispersing global pressure across all the queries, and easing the demands on the encoder. The experimental results demonstrate that MABP significantly outperforms the state-of-the-art methods in all three splits on the gRefCOCO dataset. Moreover, MABP outperforms the state-of-the-art methods on the RefCOCO+ and G-Ref datasets, and achieves very competitive results on RefCOCO.
BibTeX
@article{li2025bring,
  author = {Weize Li and Zhicheng Zhao and Haochen Bai and Fei Su},
  title = {{Bring Adaptive Binding Prototypes to Generalized Referring Expression Segmentation}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3565964},
}

Loading related content...

Figure