GRES: Generalized Referring Expression Segmentation

Chang Liu, Henghui Ding, Xudong Jiang

Referring Expression Segmentation (RES) is a fundamental task in computer vision that involves generating a binary mask for an object based on a natural language description—such as "the person in the red shirt." While researchers have made significant progress, traditional RES systems suffer from a "single-target" constraint: they assume every query refers to exactly one object.

In their paper, "GRES: Generalized Referring Expression Segmentation," Liu et al. break this constraint, moving toward a more flexible and realistic AI capable of handling multiple targets or identifying when a description matches nothing at all.

The Problem: Single-Target Limitations

Classic RES models are built on a "one expression, one target" rule. This creates two significant problems in real-world applications:

  1. Multi-target failure: If a user says "all the people at the table," a classic model might only segment one person or produce a fragmented mask.
  2. Hallucination: If asked for "the blue car" in a photo containing only red cars, a classic model—forced to find something—will likely hallucinate a mask over a red car.
Classic RES vs. GRES: Traditional models fail to segment multiple targets correctly.

The gRefCOCO Dataset

To support this new task, the authors introduced gRefCOCO, the first large-scale GRES dataset. It is a massive expansion of the popular RefCOCO dataset, containing 278,232 expressions across nearly 20,000 images.

The dataset includes:

  • Multi-target samples: Expressions like "Everyone except the kid in white" or "The two players on the left."
  • No-target samples: Expressions that intentionally refer to objects not present in the image to test the model's "rejection" capability.
Examples from the gRefCOCO dataset showing multi-target and no-target queries.

Methodology: ReLA (Relationship Modeling)

To tackle the increased complexity of GRES, the authors propose a baseline model called ReLA. Unlike standard models that perform pixel-level classification directly, ReLA focuses on regions and their relationships through two key components:

  • Region-Image Cross Attention (RIA): Uses learnable queries to dynamically aggregate features from relevant image areas into "region features."
  • Region-Language Cross Attention (RLA): This core reasoning engine explicitly models dependencies between regions and language tokens, allowing the model to understand how objects relate to each other and how specific words map to specific regions.
  • No-target Indicator: A dedicated branch predicts a binary "no-target" label, allowing the model to explicitly reject expressions that don't match the image content.

Key Results and Insights

ReLA sets new records on classic datasets (RefCOCO, RefCOCO+, G-Ref) and significantly outperforms adapted versions of existing RES models on the gRefCOCO benchmark. For example, it achieves a 63.60% gIoU (generalized IoU) on the validation set, compared to 58.40% for the previous state-of-the-art model, LAVT.

Interestingly, the authors found that training on the more difficult GRES task actually improves performance on classic single-target RES. By learning to handle "none" or "many," the model develops a deeper understanding of visual-linguistic alignment. However, identifying when an object isn't there remains a significant challenge, particularly when expressions are "deceptive" (e.g., describing a "kid in blue" when there is only a kid in purple).

Conclusion

GRES represents a major step toward more robust and reliable AI for robotics and video editing. By moving beyond the single-target assumption, researchers can build systems that better understand the complexities of human language and visual scenes.

Summary updated

Abstract

Referring Expression Segmentation (RES) aims to generate a segmentation mask for the object described by a given language expression. Existing classic RES datasets and methods commonly support single-target expressions only, i.e., one expression refers to one target object. Multitarget and no-target expressions are not considered. This limits the usage of RES in practice. In this paper, we introduce a new benchmark called Generalized Referring Expression Segmentation (GRES), which extends the classic RES to allow expressions to refer to an arbitrary number of target objects. Towards this, we construct the first largescale GRES dataset called gRefCOCO that contains multitarget, no-target, and single-target expressions. GRES and gRefCOCO are designed to be well-compatible with RES, facilitating extensive experiments to study the performance gap of the existing RES methods on the GRES task. In the experimental study, we find that one of the big challenges of GRES is complex relationship modeling. Based on this, we propose a region-based GRES baseline ReLA that adaptively divides the image into regions with subinstance clues, and explicitly models the region-region and region-language dependencies. The proposed approach ReLA achieves new state-of-the-art performance on the both newly proposed GRES and classic RES tasks. The proposed gRefCOCO dataset and method are available at https://henghuiding.github.io/GRES.
BibTeX
@inproceedings{liu2023gres,
  author = {Chang Liu and Henghui Ding and Xudong Jiang},
  title = {{GRES: Generalized Referring Expression Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure