RESMatch: Referring expression segmentation in a semi-supervised manner

Ying Zang, Runlong Cao, Chenglong Fu, Didi Zhu, Min Zhang, Wenjun Hu, Lanyun Zhu, Tianrun Chen

Introduction

Referring Expression Segmentation (RES) is a sophisticated computer vision task that requires a model to identify and segment specific objects in an image based on natural language descriptions (e.g., "the person in the blue shirt standing next to the tree"). Unlike standard semantic segmentation, which operates on a fixed set of categories, RES demands a deep, fine-grained understanding of both visual context and free-form linguistic relationships.

The primary bottleneck for RES is the high cost of data acquisition. Pixel-level annotation is notoriously expensive and time-consuming. While Semi-Supervised Learning (SSL) has been highly effective for image classification, standard methods like FixMatch often fail in RES because their data augmentations—such as random cropping or color jittering—can break the critical relationships described in the text prompt. RESMatch addresses this by introducing a multimodal-aware SSL framework.

Methodology

RESMatch is built upon the "weak-to-strong" consistency paradigm, where a model's prediction on a weakly augmented image serves as a pseudo-label to supervise its prediction on a strongly augmented version. To adapt this for the multimodal nature of RES, the authors introduced three key innovations:

  1. Spatial-Preserving Image Augmentation: Standard perturbations are replaced with a curated set of augmentations that preserve spatial and color semantics. For instance, if an image is flipped horizontally, the text prompt is automatically updated (e.g., "man on the left" becomes "man on the right").
  2. BERT-Filtered Text Augmentation (TA): To improve linguistic robustness, NLP techniques like synonym replacement and random insertion/deletion are applied to text queries. To ensure these variations remain meaningful, BERT embeddings are used to calculate cosine similarity, filtering out any augmented descriptions that stray too far from the original intent.
  3. Model Adaptive Guidance (MAG): Recognizing that not all pseudo-labels are reliable, RESMatch calculates a mask-aware confidence score. If the model is uncertain about a pseudo-label, MAG automatically reduces the augmentation strength for that sample, preventing the model from learning from noisy data early in training.
The RESMatch framework architecture featuring text augmentation and model-adaptive guidance.

Experimental Results

The model was evaluated on standard benchmarks including RefCOCO, RefCOCO+, and RefCOCOg under various labeling ratios (0.1% to 10%).

Key Findings

  • State-of-the-Art Performance: RESMatch consistently outperformed supervised-only baselines and existing SSL methods like FixMatch and CCVC. On RefCOCO with only 1% labeled data, RESMatch improved the overall Intersection over Union (oIoU) by +8.84% over the supervised baseline.
  • Efficiency: With only 10% of labeled data, RESMatch achieves approximately 87% of the performance of a model trained on the full 100% dataset, demonstrating significant diminishing returns for exhaustive manual annotation.
  • Qualitative Superiority: Visualizations demonstrate that RESMatch is significantly better at localizing targets and defining precise boundaries compared to standard models, which often produce fragmented masks when data is scarce.
Qualitative comparison between Ground Truth (GT), FixMatch, and RESMatch across different scenarios.

Conclusion and Future Work

RESMatch proves that SSL for multimodal tasks requires a specialized approach that respects the "locked pair" relationship between image and text. By using adaptive supervision and semantic-preserving augmentations, it effectively breaks the data barrier in RES.

Despite its success, the model still faces challenges with highly abstract reasoning and ambiguous scenes containing many similar objects. Future research may explore the integration of Large Language Models (LLMs) to enhance complex reasoning or more sophisticated latent space alignment between vision and language features.

Summary updated

BibTeX
@inproceedings{zang2024resmatch,
  author = {Ying Zang and Runlong Cao and Chenglong Fu and Didi Zhu and Min Zhang and Wenjun Hu and Lanyun Zhu and Tianrun Chen},
  title = {{RESMatch: Referring expression segmentation in a semi-supervised manner}},
  booktitle = {Information},
  year = {2024},
  doi = {10.1016/j.ins.2024.121709},
}
Figure