RefTeacher: A Strong Baseline for Semi-Supervised Referring Expression Comprehension

Jiamu Sun, Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, Rongrong Ji

Abstract

Referring Expression Comprehension (REC) is a challenging task that requires precise localization of objects based on natural language descriptions. Traditional REC models rely heavily on large-scale, human-annotated datasets, which are expensive to produce. We present RefTeacher, the first dedicated framework for semi-supervised REC. By employing a teacher-student paradigm enhanced with Attention-based Imitation Learning (AIL) and Adaptive Pseudo-label Weighting (APW), RefTeacher effectively leverages unlabeled data. Our experiments show that RefTeacher achieves nearly 100% of the performance of fully supervised models using only 10% of the labeled data, setting a strong baseline for future research in semi-supervised vision-language tasks.

Introduction

Referring Expression Comprehension (REC) involves locating a specific object in an image given a natural language query (e.g., "the woman in the blue hat"). Unlike standard object detection, REC requires a deep understanding of spatial relations and object attributes. However, obtaining bounding box annotations for every image-text pair is a significant bottleneck.

While semi-supervised learning (SSL) has been successful in general object detection, it faces two primary "task gaps" in REC:

  1. Sparse Supervision: REC typically provides only one bounding box per expression, resulting in a thin supervision signal compared to multi-label detection tasks.
  2. Pseudo-label Noise: Standard SSL methods often discard low-confidence pseudo-labels. In REC, where exactly one box must be predicted for a query, hard filtering can lead to the loss of valuable training samples if the prediction is even slightly inaccurate.

Methodology

RefTeacher addresses these gaps through a teacher-student framework. A Teacher network, updated via Exponential Moving Average (EMA), generates pseudo-labels on unlabeled data to guide a Student network.

The RefTeacher framework architecture featuring Teacher and Student networks with Attention-based Imitation Learning (AIL) and Adaptive Pseudo-label Weighting (APW).

Attention-based Imitation Learning (AIL)

To overcome sparse supervision, AIL forces the student to imitate the teacher's internal attention maps. While the final output (the box) is sparse, the attention mechanism provides a dense signal that teaches the model how to align visual features with linguistic cues.

Adaptive Pseudo-label Weighting (APW)

Rather than using hard thresholds to discard noisy pseudo-labels, RefTeacher employs APW. This mechanism uses the teacher's confidence score to softly weight the loss. If the teacher is uncertain, the student's learning from that sample is down-weighted, mitigating confirmation bias without losing the context of unique linguistic queries.

Experimental Results

We evaluated RefTeacher on gold-standard benchmarks including RefCOCO, RefCOCO+, and RefCOCOg. The framework demonstrated remarkable data efficiency and performance gains over existing baselines.

Comparison of accuracy over training steps for Fully Supervised, STAC, and RefTeacher models on a limited labeled data subset.

Key findings include:

  • Efficiency: With only 10% of labeled data, RefTeacher reached within 2.78% of the performance of a 100% supervised model on RefCOCO.
  • Performance Gains: Compared to a purely supervised baseline using 0.1% of data, RefTeacher improved accuracy by +19.5%.
  • Specialization: RefTeacher significantly outperformed STAC, a state-of-the-art semi-supervised detection method, proving that REC requires specialized SSL strategies.

Conclusion

RefTeacher provides a robust solution for semi-supervised REC by bridging the gap between sparse supervision and noisy pseudo-labels. By supervising the "process" (attention) rather than just the "result" (bounding box), it enables models to learn effectively from unlabeled data. Future work will focus on improving performance in highly occluded scenes and integrating larger-scale vision-language pre-trained models to further reduce annotation requirements.

Summary updated

Abstract

Referring expression comprehension (REC) often requires a large number of instance-level annotations for fully supervised learning, which are laborious and expensive. In this paper, we present the first attempt of semi-supervised learning for REC and propose a strong baseline method called RefTeacher. Inspired by the recent progress in computer vision, RefTeacher adopts a teacher-student learning paradigm, where the teacher REC network predicts pseudolabels for optimizing the student one. This paradigm allows REC models to exploit massive unlabeled data based on a small fraction of labeled. In particular, we also identify two key challenges in semi-supervised REC, namely, sparse supervision signals and worse pseudo-label noise. To address these issues, we equip RefTeacher with two novel designs called Attention-based Imitation Learning (AIL) and Adaptive Pseudo-label Weighting (APW). AIL can help the student network imitate the recognition behaviors of the teacher, thereby obtaining sufficient supervision signals. APW can help the model adaptively adjust the contributions of pseudo-labels with varying qualities, thus avoiding confirmation bias. To validate RefTeacher, we conduct extensive experiments on three REC benchmark datasets. Experimental results show that RefTeacher obtains obvious gains over the fully supervised methods. More importantly, using only 10% labeled data, our approach allows the model to achieve near 100% fully supervised performance, e.g., only −2.78% on RefCoco. Project: https://refteacher.github.io/.
BibTeX
@inproceedings{sun2023refteacher,
  author = {Jiamu Sun and Gen Luo and Yiyi Zhou and Xiaoshuai Sun and Guannan Jiang and Zhiyu Wang and Rongrong Ji},
  title = {{RefTeacher: A Strong Baseline for Semi-Supervised Referring Expression Comprehension}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01835},
}
Figure