Abstract
Referring Expression Comprehension (REC) is a challenging task that requires precise localization of objects based on natural language descriptions. Traditional REC models rely heavily on large-scale, human-annotated datasets, which are expensive to produce. We present RefTeacher, the first dedicated framework for semi-supervised REC. By employing a teacher-student paradigm enhanced with Attention-based Imitation Learning (AIL) and Adaptive Pseudo-label Weighting (APW), RefTeacher effectively leverages unlabeled data. Our experiments show that RefTeacher achieves nearly 100% of the performance of fully supervised models using only 10% of the labeled data, setting a strong baseline for future research in semi-supervised vision-language tasks.
Introduction
Referring Expression Comprehension (REC) involves locating a specific object in an image given a natural language query (e.g., "the woman in the blue hat"). Unlike standard object detection, REC requires a deep understanding of spatial relations and object attributes. However, obtaining bounding box annotations for every image-text pair is a significant bottleneck.
While semi-supervised learning (SSL) has been successful in general object detection, it faces two primary "task gaps" in REC:
- Sparse Supervision: REC typically provides only one bounding box per expression, resulting in a thin supervision signal compared to multi-label detection tasks.
- Pseudo-label Noise: Standard SSL methods often discard low-confidence pseudo-labels. In REC, where exactly one box must be predicted for a query, hard filtering can lead to the loss of valuable training samples if the prediction is even slightly inaccurate.
Methodology
RefTeacher addresses these gaps through a teacher-student framework. A Teacher network, updated via Exponential Moving Average (EMA), generates pseudo-labels on unlabeled data to guide a Student network.
Attention-based Imitation Learning (AIL)
To overcome sparse supervision, AIL forces the student to imitate the teacher's internal attention maps. While the final output (the box) is sparse, the attention mechanism provides a dense signal that teaches the model how to align visual features with linguistic cues.
Adaptive Pseudo-label Weighting (APW)
Rather than using hard thresholds to discard noisy pseudo-labels, RefTeacher employs APW. This mechanism uses the teacher's confidence score to softly weight the loss. If the teacher is uncertain, the student's learning from that sample is down-weighted, mitigating confirmation bias without losing the context of unique linguistic queries.
Experimental Results
We evaluated RefTeacher on gold-standard benchmarks including RefCOCO, RefCOCO+, and RefCOCOg. The framework demonstrated remarkable data efficiency and performance gains over existing baselines.
Key findings include:
- Efficiency: With only 10% of labeled data, RefTeacher reached within 2.78% of the performance of a 100% supervised model on RefCOCO.
- Performance Gains: Compared to a purely supervised baseline using 0.1% of data, RefTeacher improved accuracy by +19.5%.
- Specialization: RefTeacher significantly outperformed STAC, a state-of-the-art semi-supervised detection method, proving that REC requires specialized SSL strategies.
Conclusion
RefTeacher provides a robust solution for semi-supervised REC by bridging the gap between sparse supervision and noisy pseudo-labels. By supervising the "process" (attention) rather than just the "result" (bounding box), it enables models to learn effectively from unlabeled data. Future work will focus on improving performance in highly occluded scenes and integrating larger-scale vision-language pre-trained models to further reduce annotation requirements.
Summary updated