Differentiated Relevances Embedding for Group-based Referring Expression Comprehension

Fuhai Chen, Xiaoshuai Sun, Xuri Ge, Jianzhuang Liu, Yongjian Wu, Feiyue Huang, Rongrong Ji

Abstract

Referring Expression Comprehension (REC) involves localizing a target object in an image based on a natural language description. Traditional REC training often suffers from a lack of diverse negative samples and the difficulty of managing sample 'hardness' during training. This paper introduces Group-based REC, a framework that leverages semantically similar images to provide a richer set of negative examples. We propose Multi-group Self-paced Relevance Learning (MSRL), which adaptively prioritizes training samples based on cross-modal relevance and ensures balanced learning across different object categories. Experimental results on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that MSRL achieves state-of-the-art performance.

Introduction

Referring Expression Comprehension (REC) is a critical task at the intersection of computer vision and natural language processing. The goal is to identify a specific object (the anchor) in an image given a textual expression. Standard training paradigms use triplet loss, comparing a positive pair (anchor object and its description) against negative pairs (the same description paired with other objects in the same image).

However, this traditional approach faces two significant hurdles:

  1. Limited Negatives: A single image typically contains few objects of the same category, limiting the model's exposure to fine-grained attribute differences (e.g., distinguishing between different shades of a color).
  2. The Hardness Problem: Not all negative samples are equally informative. Some are too easy, while others are overly complex for early training stages. Additionally, different object categories (e.g., 'people' vs. 'cars') exhibit varying levels of inherent similarity, which can bias the learning process if not properly regularized.

Proposed Method: Group-based REC and MSRL

To overcome these limitations, the authors propose a group-based data structure and a self-paced learning schema.

Comparison between (a) Traditional REC and (b) the proposed Group-based REC. Grouping images by category allows for a wider variety of learnable attributes and more challenging negative samples.

Multi-group Self-paced Relevance Learning (MSRL)

The MSRL framework consists of three core components:

  • Group-based Data Structure: Images are organized into groups based on object categories. This allows the model to draw negative samples from different images within the same group, providing a much broader spectrum of attribute values for the model to learn.
  • Within-group Adaptive Learning: Following the principles of curriculum learning, the model initially focuses on 'easy' negatives (low cross-modal relevance). As training progresses, it adaptively lowers the threshold to include 'harder' negatives, refining its ability to distinguish subtle differences.
  • Across-group Relevance Constraint: To prevent overfitting to specific categories, an $l_{F,1}$-norm regularization term is introduced. This ensures that the learning pace is balanced across different object groups, accounting for the fact that some categories are naturally more similar than others.

Experimental Results

The MSRL method was evaluated on three major benchmarks: RefCOCO, RefCOCO+, and RefCOCOg. The model consistently outperformed existing state-of-the-art methods:

  • RefCOCOg: MSRL achieved 79.15% accuracy on the validation set and 78.42% on the test set using ground-truth regions, surpassing the previous baseline (MAttN).
  • RefCOCO+: On the TestB set, the model reached 66.47%, outperforming Multi-hop FiLM.
  • Robustness: The framework maintained high performance even when using automatically detected object regions, demonstrating its resilience to noise.

Conclusion

By rethinking the structure of negative samples and the pace of learning, the MSRL framework significantly advances the state-of-the-art in Referring Expression Comprehension. The transition from image-based to group-based learning allows for a more robust understanding of object attributes. Future work may explore automated category discovery and the application of these principles to related tasks like person re-identification.

Summary updated

Abstract

Referring expression comprehension (REC) aims to locate a certain object in an image referred by a natural language expression. For joint understanding of regions and expressions, existing REC works typically target on modeling the cross-modal relevance in each region-expression pair within each single image. In this paper, we explore a new but general REC-related problem, named Group-based REC, where the regions and expressions can come from different subject-related images (images in the same group), e.g., sets of photo albums or video frames. Different from REC, Group-based REC involves differentiated cross-modal relevances within each group and across different groups, which, however, are neglected in the existing one-line paradigm. To this end, we propose a novel relevance-guided multi-group self-paced learning schema (termed RMSL), where the within-group region-expression pairs are adaptively assigned with different priorities according to their cross-modal relevances, and the bias of the group priority is balanced via an across-group relevance constraint simultaneously. In particular, based on the visual and textual semantic features, RMSL conducts an adaptive learning cycle upon triplet ranking, where (1) the target-negative region-expression pairs with low within-group relevances are used preferentially in model training to distinguish the primary semantics of the target objects, and (2) an across-group relevance regularization is integrated into model training to balance the bias of group priority. The relevances, the pairs, and the model parameters are alternatively updated upon a unified self-paced hinge loss.
BibTeX
@misc{chen2022differentiated,
  author = {Fuhai Chen and Xiaoshuai Sun and Xuri Ge and Jianzhuang Liu and Yongjian Wu and Feiyue Huang and Rongrong Ji},
  title = {{Differentiated Relevances Embedding for Group-based Referring Expression Comprehension}},
  year = {2022},
  eprint = {2203.06382},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.06382},
}
Figure