Abstract
Referring Expression Comprehension (REC) involves localizing a target object in an image based on a natural language description. Traditional REC training often suffers from a lack of diverse negative samples and the difficulty of managing sample 'hardness' during training. This paper introduces Group-based REC, a framework that leverages semantically similar images to provide a richer set of negative examples. We propose Multi-group Self-paced Relevance Learning (MSRL), which adaptively prioritizes training samples based on cross-modal relevance and ensures balanced learning across different object categories. Experimental results on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that MSRL achieves state-of-the-art performance.
Introduction
Referring Expression Comprehension (REC) is a critical task at the intersection of computer vision and natural language processing. The goal is to identify a specific object (the anchor) in an image given a textual expression. Standard training paradigms use triplet loss, comparing a positive pair (anchor object and its description) against negative pairs (the same description paired with other objects in the same image).
However, this traditional approach faces two significant hurdles:
- Limited Negatives: A single image typically contains few objects of the same category, limiting the model's exposure to fine-grained attribute differences (e.g., distinguishing between different shades of a color).
- The Hardness Problem: Not all negative samples are equally informative. Some are too easy, while others are overly complex for early training stages. Additionally, different object categories (e.g., 'people' vs. 'cars') exhibit varying levels of inherent similarity, which can bias the learning process if not properly regularized.
Proposed Method: Group-based REC and MSRL
To overcome these limitations, the authors propose a group-based data structure and a self-paced learning schema.
Multi-group Self-paced Relevance Learning (MSRL)
The MSRL framework consists of three core components:
- Group-based Data Structure: Images are organized into groups based on object categories. This allows the model to draw negative samples from different images within the same group, providing a much broader spectrum of attribute values for the model to learn.
- Within-group Adaptive Learning: Following the principles of curriculum learning, the model initially focuses on 'easy' negatives (low cross-modal relevance). As training progresses, it adaptively lowers the threshold to include 'harder' negatives, refining its ability to distinguish subtle differences.
- Across-group Relevance Constraint: To prevent overfitting to specific categories, an $l_{F,1}$-norm regularization term is introduced. This ensures that the learning pace is balanced across different object groups, accounting for the fact that some categories are naturally more similar than others.
Experimental Results
The MSRL method was evaluated on three major benchmarks: RefCOCO, RefCOCO+, and RefCOCOg. The model consistently outperformed existing state-of-the-art methods:
- RefCOCOg: MSRL achieved 79.15% accuracy on the validation set and 78.42% on the test set using ground-truth regions, surpassing the previous baseline (MAttN).
- RefCOCO+: On the TestB set, the model reached 66.47%, outperforming Multi-hop FiLM.
- Robustness: The framework maintained high performance even when using automatically detected object regions, demonstrating its resilience to noise.
Conclusion
By rethinking the structure of negative samples and the pace of learning, the MSRL framework significantly advances the state-of-the-art in Referring Expression Comprehension. The transition from image-based to group-based learning allows for a more robust understanding of object attributes. Future work may explore automated category discovery and the application of these principles to related tasks like person re-identification.
Summary updated