Abstract
Discovering social relations in images can make machines better interpret the behavior of human beings. However, automatically recognizing social relations in images is a challenging task due to the significant gap between the domains of visual content and social relation. Existing studies separately process various features such as faces expressions, body appearance, and contextual objects, thus they cannot comprehensively capture the multi-granularity semantics, such as scenes, regional cues of persons, and interactions among persons and objects. To bridge the domain gap, we propose a Multi-Granularity Reasoning framework for social relation recognition from images. The global knowledge and mid-level details are learned from the whole scene and the regions of persons and objects, respectively. Most importantly, we explore the fine-granularity pose keypoints of persons to discover the interactions among persons and objects. Specifically, the pose-guided Person-Object Graph and Person-Pose Graph are proposed to model the actions from persons to object and the interactions between paired persons, respectively. Based on the graphs, social relation reasoning is performed by graph convolutional networks. Finally, the global features and reasoned knowledge are integrated as a comprehensive representation for social relation recognition. Extensive experiments on two public datasets show the effectiveness of the proposed framework.
Introduction
While computer vision has reached high proficiency in object identification, recognizing the abstract nature of human connections—social relations—remains a significant challenge. This "domain gap" exists because visual pixels do not directly map to social semantics. For instance, identifying two individuals is straightforward, but determining if they are family, friends, or strangers requires a deeper understanding of their environment and interactions. Existing methods often fail by analyzing features like facial expressions or body appearance in isolation. This paper introduces a Multi-Granularity Reasoning (MGR) framework that bridges this gap by integrating global scene context with fine-grained interaction modeling.
Methodology
The MGR framework processes images through a dual-branch architecture to capture semantics at different levels of granularity.
Global Knowledge Extraction
A deep CNN (ResNet-101) extracts features from the entire image to capture "scene-level" knowledge. This is particularly effective for identifying non-intimate relations (e.g., Professional or Commercial) which are often tied to specific settings like offices or shops.
Regional and Interaction Reasoning
The core of the MGR framework lies in its ability to model interactions through two specialized graphs:
- Person-Object Graph (POG): This graph establishes functional connections between people and surrounding objects. By using pose keypoints (e.g., hands), the model determines if a person is actively interacting with an object rather than just being near it.
- Person-Pose Graph (PPG): This graph models physical interactions between pairs of people using 17 skeleton keypoints. It distinguishes between active nodes (wrists, ankles) and passive nodes to capture gestures like hand-holding or walking in tandem.
These graphs are processed by Graph Convolutional Networks (GCNs) to perform relational reasoning. The final prediction is a fusion of scores from the global CNN and the GCN-based interaction branches.
Experiments and Results
The framework was evaluated on two major benchmarks: the People in Social Context (PISC) dataset and the People in Photo Album (PIPA) dataset.
Performance Comparison
MGR achieved state-of-the-art results on both datasets. On PISC, it reached a mean Average Precision (mAP) of 70.0%, outperforming the previous best model (GRM). On PIPA, it achieved 64.4% accuracy.
Key Insights
Ablation studies confirmed that pose guidance is critical. Incorporating pose information into the Person-Object Graph improved performance, demonstrating that understanding how a person interacts with an object is more informative than simple proximity. Furthermore, the study revealed that intimate relations (Family, Friends) rely more on fine-grained pose interactions, while professional relations are better identified through global scene context.
Conclusion
The Multi-Granularity Reasoning framework successfully bridges the gap between visual content and social semantics by combining global context with fine-grained pose-guided interaction modeling. While social relation recognition remains challenging due to category ambiguity and data imbalance, MGR sets a new standard for accuracy. Future work will focus on enhancing context cues to better handle rare social categories.
Summary updated