CPARR: Category-based Proposal Analysis for Referring Relationships

Chuanzi He, Haidong Zhu, Jiyang Gao, Kan Chen, Ram Nevatia

Abstract

The task of referring relationships is to localize subject and object entities in an image satisfying a relationship query, given in the form of <subject, predicate, object>. This requires simultaneous localization of both entities within a specified interaction. We introduce CPARR, a simple yet effective proposal-based method for referring relationships. Unlike existing attention-based methods such as SSAS, CPARR generates high-resolution results while reducing complexity and ambiguity. Our framework consists of two modules: a category-based proposal generation module to select relevant entity candidates and a predicate analysis module to score the compatibility of candidate pairs. CPARR achieves state-of-the-art performance on the Visual Relationship Detection (VRD) and Visual Genome (VG) datasets.

Introduction

Referring relationships represent a significant step beyond standard object detection. While noun phrase grounding asks a model to find a "red car," referring relationships require identifying two entities and verifying a specific interaction between them, such as "the car parked next to the building."

Previous research in this area has faced two major hurdles. First, many state-of-the-art methods rely on low-resolution attention maps (e.g., 14x14 pixels), which limits the precision of generated bounding boxes. Second, general Visual Relationship Detection (VRD) models often struggle with specific queries because they attempt to identify all relationships in an image simultaneously, leading to computational inefficiency and a bias toward common relationships over the specific ones requested.

Methodology

CPARR (Category-based Proposal Analysis for Referring Relationships) addresses these issues by decoupling entity localization from relationship verification.

Category-based Proposal Generation

The model first employs a standard Region Proposal Network (RPN) to generate candidate boxes. Two independent modules ($M_{sub}$ and $M_{obj}$) then score and refine these boxes based solely on the categories in the query (e.g., "person" and "bag"). This narrows the search space to a few high-quality candidates for each role, avoiding the ambiguity of searching for the relationship immediately.

Predicate Analysis Module

The model takes the top-K subject and object candidates and evaluates all possible pairs. For each pair, spatial feature maps are extracted using ROI pooling. Unlike previous methods that used flattened 1D vectors, CPARR preserves spatial information by concatenating 3D feature maps. A small CNN then scores these pairs to determine which best satisfies the predicate (e.g., "next to" or "riding"). The final prediction combines category scores with the predicate score to ensure both entity type and relationship accuracy.

Qualitative results of CPARR showing (a, b) category-based proposals and (c, d) final referring relationship localizations.

Experimental Results

The model was evaluated on the VRD and Visual Genome (VG) datasets, both known for high levels of entity ambiguity.

Quantitative Performance

CPARR demonstrated a clear lead over existing baselines:

  • Mean IoU: On the VRD dataset, CPARR achieved a Mean IoU of 0.482 for subjects and 0.510 for objects, significantly outperforming the previous best (SSAS) at 0.369 and 0.410, respectively.
  • Recall@1: CPARR nearly doubled the performance of SSAS in top-1 recall on the VRD dataset (0.498 vs 0.215 for subjects).

Disambiguation and Efficiency

Qualitative results show that CPARR excels in complex scenes. By decoupling the tasks, the model avoids the "blurriness" associated with end-to-end attention models. Furthermore, by analyzing only the top $5 \times 5$ pairs, CPARR drastically reduces computational overhead compared to dense relationship detection methods.

Conclusion

CPARR introduces a robust framework for referring relationships by separating localization from predicate verification. This approach allows for high-resolution localization and effective disambiguation in complex visual scenes. While the predicate module remains a focus for future improvement—particularly for visually subtle interactions—the current results set a new benchmark for precision and efficiency in the field.

Summary updated

Abstract

The task of referring relationships is to localize subject and object entities in an image satisfying a relationship query, which is given in the form of \texttt{<subject, predicate, object>}. This requires simultaneous localization of the subject and object entities in a specified relationship. We introduce a simple yet effective proposal-based method for referring relationships. Different from the existing methods such as SSAS, our method can generate a high-resolution result while reducing its complexity and ambiguity. Our method is composed of two modules: a category-based proposal generation module to select the proposals related to the entities and a predicate analysis module to score the compatibility of pairs of selected proposals. We show state-of-the-art performance on the referring relationship task on two public datasets: Visual Relationship Detection and Visual Genome.
BibTeX
@inproceedings{he2020cparr,
  author = {Chuanzi He and Haidong Zhu and Jiyang Gao and Kan Chen and Ram Nevatia},
  title = {{CPARR: Category-based Proposal Analysis for Referring Relationships}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=apT2Qy2skG2},
}
Figure