TL;DR
This paper introduces MABP (Model with Adaptive Binding Prototypes), a novel framework designed for Generalized Referring Expression Segmentation (GRES). By partitioning images into spatial regions and "binding" specific language-aware queries to those regions, the model effectively handles complex natural language prompts that describe multiple objects, different categories, or even no objects at all.
What Problem Does This Paper Address?
Traditional Referring Expression Segmentation (RES) operates on a simple rule: one sentence describes exactly one object. However, real-world scenarios are rarely that tidy. Generalized RES (GRES) was introduced to handle expressions that refer to multiple targets (e.g., "the two dogs and the man") or no targets at all (e.g., "the cat" when no cat is present).
Existing RES models often use a "greedy" approach, where a set of global query vectors tries to summarize all foreground objects into a single class prototype. This fails in GRES because:
- Category Conflict: A single prototype struggles to represent multiple distinct categories (like a "dog" and a "guy") simultaneously.
- No-Target Instability: When no target exists, the attention mechanisms in these models become unstable, often leading to "hallucinated" segmentations.
- Encoder Pressure: Forcing the encoder to map diverse objects into a single feature representation places an immense burden on the backbone network.
Methods
The authors propose MABP, which shifts from global prototypes to Adaptive Binding Prototypes. The architecture consists of three main innovations:
- Region-Text Specific Query Generator: Instead of random initialization, the model uses learnable region embeddings. These are combined with linguistic features via cross-attention to create initial queries that already "know" which part of the image they are responsible for and what they are looking for.
- Mixed Modal Decoder (MMD): To handle "no-target" samples, the decoder doesn't just look at visual features. It explicitly incorporates linguistic features as "placeholders" during the self-attention phase. This ensures that even if the image contains no targets, the attention map remains stable and manageable, leading to a clean "empty" prediction.
- Regional Supervision Head (RSH): This is the core "binding" mechanism. The model partitions the feature map into a grid of patches. Each query is strictly supervised by the ground truth of its corresponding patch. This forces queries to specialize in local details and prevents the "global pressure" found in previous models.
Datasets
The model was evaluated on several key benchmarks:
- gRefCOCO: The primary large-scale dataset for GRES, containing multi-target and no-target expressions.
- RefCOCO / RefCOCO+ / G-Ref: Standard RES datasets used to test the model's generalization to classic single-target tasks.
- MeViS: A large-scale benchmark for referring video object segmentation, used to test the upper limits of the model's flexibility.
Key Insights and Ideas
- Divide and Conquer: By forcing queries to bind to specific spatial regions, the model allows different queries to represent different categories (e.g., one query handles the "dog" in the bottom-left while another handles the "man" in the center).
- Stability through Language: Inverting the typical attention flow by using language features as anchors in the decoder significantly improves performance on "empty" samples. It prevents the model from "searching" too hard for a target that isn't there.
- Preserving Pre-trained Knowledge: Standard GRES training often forces encoders to map different categories into the same "foreground" feature space. MABP’s regional approach allows the encoder to maintain its pre-trained ability to distinguish between classes, as the "merging" happens later in the regional heads.
Key Results
MABP achieved state-of-the-art (SOTA) results across multiple benchmarks:
- gRefCOCO: Outperformed the previous best model (ReLA) by approximately 2% in cIoU and 3% in gIoU across all splits.
- No-Target Performance: On the difficult "no-target" samples, MABP improved accuracy (N-acc) from 57.51% to 64.55%, a significant jump in reliability.
- Precision at High Thresholds: The model showed a notable improvement in [email protected] (precision at a 0.9 IoU threshold), indicating it is much better at capturing fine details and small objects than its predecessors.
- Generalization: Despite being designed for GRES, MABP also outperformed SOTA RES models on RefCOCO+ and G-Ref.
Limitations and Future Work
The authors identify two main areas for improvement:
- Rigid Partitioning: Currently, the model uses a "hard-split" strategy with fixed, coarse-grained grid boundaries. This can cause issues if a small object sits exactly on the boundary between two regions.
- Complex No-Target Scenarios: While MABP handles "obvious" no-target cases well (e.g., "laptop" when there is no laptop), it still struggles with "near-miss" cases where the expression is very similar to something in the image (e.g., "man in blue shirt" when there is a man in a purple shirt).
Future work may involve deformable partitioning (allowing regions to change shape) or adaptive merging of prototypes to handle these edge cases more gracefully.
Summary updated