Instance-Aware Generalized Referring Expression Segmentation

E-Ro Nguyen, Hieu Le, Dimitris Samaras, Michael Ryoo

Abstract

Recent works on Generalized Referring Expression Segmentation (GRES) struggle with handling complex expressions referring to multiple distinct objects. This is because these methods typically employ an end-to-end foreground-background segmentation and lack a mechanism to explicitly differentiate and associate different object instances to the text query. To this end, we propose InstAlign, a method that incorporates object-level reasoning into the segmentation process. Our model leverages both text and image inputs to extract a set of object-level tokens that capture both the semantic information in the input prompt and the objects within the image. By modeling the text-object alignment via instance-level supervision, each token uniquely represents an object segment in the image, while also aligning with relevant semantic information from the text. Extensive experiments on the gRefCOCO and Ref-ZOM benchmarks demonstrate that our method significantly advances state-of-the-art performance, setting a new standard for precise and flexible GRES.

Introduction

Referring Expression Segmentation (RES) aims to segment a single object described by a natural language prompt. However, real-world scenarios often involve Generalized RES (GRES), where expressions may refer to multiple objects (e.g., "all the dogs") or no objects at all ("no-target").

Current GRES methods often treat the task as a binary segmentation problem, producing a single foreground mask. This "blob-based" approach fails to distinguish between individual instances or link specific phrases to specific objects. For example, in the prompt "the girl holding a small dog and the dog on the right," a standard model might merge all relevant entities into one mass, losing the fine-grained relationships required for accurate segmentation in crowded scenes.

Comparison of traditional region-based methods versus the proposed instance-aware approach.

Proposed Method: InstAlign

InstAlign addresses these limitations by treating GRES as an instance-aware task. Built upon the Mask2Former architecture, it introduces several key components to align linguistic phrases with visual instances.

The InstAlign architecture, illustrating the flow from encoders to the Phrase-Object Transformer and final instance aggregation.

Phrase-Object Transformer (POT)

Unlike traditional models that use text as a static filter, the POT employs a bidirectional attention mechanism. It iteratively refines object queries (visual) and text features (linguistic), allowing the model to "ground" specific words into discrete visual instances.

Phrase-Object Alignment (POA) Loss

To enforce explicit mapping, the POA loss calculates the similarity between an object query and the specific text phrase that describes it. This supervision ensures that each visual segment is tied to its corresponding linguistic description.

Adaptive Instance Aggregation (AIA)

Rather than relying on hard confidence thresholds, AIA uses a soft-assignment approach. It weights each predicted instance mask by its relevance score before merging them into the final output. This provides robustness against ambiguity and overlapping objects.

No-Target Predictor

To handle "no-target" scenarios, the model generates a global token—a weighted sum of object and text features—to predict the likelihood that no valid targets exist in the image.

Experimental Results

InstAlign was evaluated on gRefCOCO, Ref-ZOM, and traditional RES benchmarks (RefCOCO/+/g).

  • gRefCOCO: InstAlign achieved a gIoU of 74.34%, a 3.3% improvement over previous state-of-the-art (SOTA). Notably, it reached a no-target accuracy (N-acc) of 79.72%, a 12.25% increase.
  • Ref-ZOM: The model set new records across all categories, particularly in "one-to-many" settings, outperforming the DMMI model by over 4% in mIoU.
  • Traditional RES: On RefCOCOg, InstAlign surpassed previous models like MagNet by over 5% in validation accuracy, demonstrating that instance-aware reasoning benefits even single-target tasks.

Conclusion

InstAlign demonstrates that instance-level reasoning is critical for GRES. By explicitly aligning phrases with visual objects and employing adaptive aggregation, the model handles complex multi-target and no-target scenarios with high precision. While the model currently faces challenges with complex hierarchical attributes, it sets a new benchmark for flexible and accurate referring expression segmentation.

Summary updated

Abstract

Recent works on Generalized Referring Expression Segmentation (GRES) struggle with handling complex expressions referring to multiple distinct objects. This is because these methods typically employ an end-to-end foreground-background segmentation and lack a mechanism to explicitly differentiate and associate different object instances to the text query. To this end, we propose InstAlign, a method that incorporates object-level reasoning into the segmentation process. Our model leverages both text and image inputs to extract a set of object-level tokens that capture both the semantic information in the input prompt and the objects within the image. By modeling the text-object alignment via instance-level supervision, each token uniquely represents an object segment in the image, while also aligning with relevant semantic information from the text. Extensive experiments on the gRefCOCO and Ref-ZOM benchmarks demonstrate that our method significantly advances state-of-the-art performance, setting a new standard for precise and flexible GRES.
BibTeX
@misc{nguyen2024instance,
  author = {E-Ro Nguyen and Hieu Le and Dimitris Samaras and Michael Ryoo},
  title = {{Instance-Aware Generalized Referring Expression Segmentation}},
  year = {2024},
  eprint = {2411.15087},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.15087},
}
Figure