Abstract
The task of referring relationships is to localize subject and object entities in an image satisfying a relationship query, given in the form of <subject, predicate, object>. This requires simultaneous localization of the subject and object entities in a specified relationship. We introduce CPARR, a simple yet effective proposal-based method for referring relationships. Unlike existing methods such as SSAS, our method generates high-resolution results while reducing complexity and ambiguity. Our method comprises two modules: a category-based proposal generation module to select proposals related to the entities and a predicate analysis module to score the compatibility of pairs of selected proposals. We demonstrate state-of-the-art performance on the referring relationship task using the Visual Relationship Detection (VRD) and Visual Genome datasets.
Introduction
In computer vision, "grounding" refers to the ability to link a textual description to a specific region in an image. While grounding a single noun (e.g., "find the car") is well-studied, referring relationships involve a triplet query: <subject, predicate, object> (e.g., "the bag next to the person").
The challenge is twofold:
- Ambiguity: In a crowded scene, there might be multiple instances of the same object category. The model must identify the specific pair that satisfies the predicate.
- Resolution: Previous state-of-the-art methods (like SSAS) relied on "attention maps"—heatmaps that highlight relevant areas. However, these maps are often very low resolution (e.g., $14 \times 14$), leading to blurry, imprecise bounding boxes that fail to tightly capture the target objects.
Methods
CPARR addresses these challenges by decoupling object identification from relationship analysis through a two-stage framework.
Category-based Proposal Generation
Instead of analyzing the entire image globally, CPARR utilizes a Region Proposal Network (RPN) to generate object candidates. Two independent modules, $M_{sub}$ and $M_{obj}$, score and refine these proposals based on the subject and object categories specified in the query. This stage uses GloVe embeddings for text and ResNet-50 for visual features, effectively filtering out irrelevant regions and reducing search complexity.
Predicate Analysis Module
After identifying the top-K candidates for the subject and object, this module evaluates pairs of proposals. Rather than using flattened 1D vectors, it concatenates 3D spatial feature maps from ROI pooling. This preserves geometric and contextual information, allowing the model to verify spatial relationships (e.g., "above", "next to") more accurately. The final prediction combines scores from both modules to ensure the result matches both the object categories and the specified relationship.
Key Results
CPARR significantly outperformed previous state-of-the-art methods across standard benchmarks. By moving from coarse attention grids to RPN-based proposals, the model achieves much tighter bounding boxes, reflected in a substantial jump in Mean Intersection over Union (IoU).
On the Visual Relationship Detection (VRD) dataset, CPARR achieved a Mean IoU of 0.482 for subjects and 0.510 for objects, compared to the previous best (SSAS) of 0.369 and 0.410. Recall@1 metrics also more than doubled, with CPARR reaching 0.498 (subject) and 0.524 (object). Similar performance gains were observed on the Visual Genome dataset, confirming the method's scalability to complex scenes.
Conclusion
CPARR demonstrates that decoupling category identification from relationship analysis is a powerful strategy for referring relationship tasks. While highly effective, the authors note that future work could address "long-tail" distributions of rare relationships and explore adaptive methods for weighing predicate scores. The shift from attention maps to proposal-based analysis marks a significant step toward high-precision visual grounding.
Summary updated