Referring Expression Segmentation (RES) is a critical task in computer vision that requires a model to identify and segment a specific object in an image based on a natural language description. Traditionally, this task has been extremely resource-intensive, requiring large-scale datasets with pixel-level mask annotations. This paper, presented at CVPR, introduces a more efficient "Partial-RES" training paradigm that significantly reduces the reliance on expensive mask data by leveraging abundant, cheaper bounding box annotations.
The Challenge of Data-Hungry Segmentation
Standard RES models typically employ two distinct "heads": one for object detection (bounding boxes) and another for segmentation (pixel masks). While bounding boxes are relatively easy to produce, pixel-perfect masks are slow and costly to annotate. In scenarios where mask data is scarce—such as having only 1% of the dataset annotated with masks—traditional models struggle. The mask-predicting component fails to generalize effectively, even if the detection component is highly accurate.
The authors address this by asking: Can we achieve high-quality segmentation by treating it as a sequence of points rather than a pixel grid?
Architecture: From Pixels to Points
The core of the proposed solution is built upon SeqTR, a model that unifies detection and segmentation into a single task: sequence prediction. Instead of predicting a mask grid, the model predicts a sequence of points that form the contour of the object.
As shown in the figure above, this unified representation allows the model to view "detecting a box" and "segmenting an object" as essentially the same task with different levels of detail. This shared format is the key to successfully transferring knowledge from abundant box-level data to the more complex task of segmentation.
Technical Innovations: CCTF, PMCA, and RPP
To bridge the gap between detection and segmentation under limited supervision, the researchers introduced three primary innovations:
- Co-Content Teacher-Forcing (CCTF): In standard sequence training, a model is given the correct previous coordinate to predict the next. CCTF enhances this by also providing the specific visual features (pixels/regions) associated with those coordinates. This forces the model to explicitly ground spatial coordinates in visual content, reducing "exposure bias" where small errors during inference lead to catastrophic failures.
- Point-Modulated Cross-Attention (PMCA): This mechanism uses a Gaussian-like prior to constrain the model’s attention. Instead of processing the entire image for every point, the model focuses on the specific region relevant to the points it is currently predicting, improving spatial accuracy.
- Resampling Pseudo Points (RPP) Strategy: To utilize the 90-99% of data lacking masks, the model generates "pseudo-masks." The RPP strategy uses ground-truth bounding boxes as a sanity check: if a generated mask does not fit well within the known box (measured by a Dice coefficient), it is discarded. This prevents the model from learning from its own "hallucinations."
Experimental Results
The model was evaluated on standard benchmarks including RefCOCO, RefCOCO+, and RefCOCOg. The researchers simulated a "Partial-RES" environment by limiting mask annotations to 1%, 5%, and 10% of the training sets.
Key findings include:
- High Efficiency: With only 10% of mask data, the model achieves 97% of the performance of a fully supervised model on RefCOCO+.
- Robustness at 1%: Even with just 1% of mask data, the model reached 52.06% accuracy on RefCOCO+@testA, compared to 58.93% for the fully supervised version.
- Superiority: The method consistently outperformed the standard SeqTR baseline by an average of 3.0% across all data-scarce settings.
Conclusion
By treating segmentation as a point-by-point sequence task and introducing mechanisms to ground these points in visual features, this research demonstrates that high-quality RES models can be trained with 100x fewer pixel-level masks. While point-based contours may currently lack the extreme smoothness of pixel-grid models, the efficiency gains offer a promising path for scaling referring expression models to larger, more diverse datasets where full mask annotation is impractical.
Summary updated