Abstract
The task of referring relationships is to localize subject and object entities in an image satisfying a relationship query, given in the form of <subject, predicate, object>. This requires simultaneous localization of both entities within a specified interaction. We introduce CPARR, a simple yet effective proposal-based method for referring relationships. Unlike existing attention-based methods such as SSAS, CPARR generates high-resolution results while reducing complexity and ambiguity. Our framework consists of two modules: a category-based proposal generation module to select relevant entity candidates and a predicate analysis module to score the compatibility of candidate pairs. CPARR achieves state-of-the-art performance on the Visual Relationship Detection (VRD) and Visual Genome (VG) datasets.
Introduction
Referring relationships represent a significant step beyond standard object detection. While noun phrase grounding asks a model to find a "red car," referring relationships require identifying two entities and verifying a specific interaction between them, such as "the car parked next to the building."
Previous research in this area has faced two major hurdles. First, many state-of-the-art methods rely on low-resolution attention maps (e.g., 14x14 pixels), which limits the precision of generated bounding boxes. Second, general Visual Relationship Detection (VRD) models often struggle with specific queries because they attempt to identify all relationships in an image simultaneously, leading to computational inefficiency and a bias toward common relationships over the specific ones requested.
Methodology
CPARR (Category-based Proposal Analysis for Referring Relationships) addresses these issues by decoupling entity localization from relationship verification.
Category-based Proposal Generation
The model first employs a standard Region Proposal Network (RPN) to generate candidate boxes. Two independent modules ($M_{sub}$ and $M_{obj}$) then score and refine these boxes based solely on the categories in the query (e.g., "person" and "bag"). This narrows the search space to a few high-quality candidates for each role, avoiding the ambiguity of searching for the relationship immediately.
Predicate Analysis Module
The model takes the top-K subject and object candidates and evaluates all possible pairs. For each pair, spatial feature maps are extracted using ROI pooling. Unlike previous methods that used flattened 1D vectors, CPARR preserves spatial information by concatenating 3D feature maps. A small CNN then scores these pairs to determine which best satisfies the predicate (e.g., "next to" or "riding"). The final prediction combines category scores with the predicate score to ensure both entity type and relationship accuracy.
Experimental Results
The model was evaluated on the VRD and Visual Genome (VG) datasets, both known for high levels of entity ambiguity.
Quantitative Performance
CPARR demonstrated a clear lead over existing baselines:
- Mean IoU: On the VRD dataset, CPARR achieved a Mean IoU of 0.482 for subjects and 0.510 for objects, significantly outperforming the previous best (SSAS) at 0.369 and 0.410, respectively.
- Recall@1: CPARR nearly doubled the performance of SSAS in top-1 recall on the VRD dataset (0.498 vs 0.215 for subjects).
Disambiguation and Efficiency
Qualitative results show that CPARR excels in complex scenes. By decoupling the tasks, the model avoids the "blurriness" associated with end-to-end attention models. Furthermore, by analyzing only the top $5 \times 5$ pairs, CPARR drastically reduces computational overhead compared to dense relationship detection methods.
Conclusion
CPARR introduces a robust framework for referring relationships by separating localization from predicate verification. This approach allows for high-resolution localization and effective disambiguation in complex visual scenes. While the predicate module remains a focus for future improvement—particularly for visually subtle interactions—the current results set a new benchmark for precision and efficiency in the field.
Summary updated