Multi-Granularity Reasoning for Social Relation Recognition From Images

Meng Zhang, Xinchen Liu, Wu Liu, Anfu Zhou, Huadong Ma, Tao Mei

TL;DR

Recognizing social relationships in photos (e.g., "Are they family or strangers?") is difficult because visual pixels don't easily map to abstract human bonds. This paper introduces a Multi-Granularity Reasoning (MGR) framework that combines global scene context, regional object data, and fine-grained body poses using Graph Convolutional Networks (GCNs) to achieve state-of-the-art accuracy in social relation recognition.

What Problem Does This Paper Address?

While AI has become excellent at identifying what is in an image (a car, a person, a dog), it struggles to understand the nature of the connections between people. This is known as the "domain gap" between visual content and social semantics.

Existing methods often look at features in isolation—like facial expressions or body appearance—but fail to capture the full picture. For instance, two people standing in a kitchen (scene) holding a grocery bag (object interaction) while standing close together (pose) are likely family. To solve this, the authors argue we need to reason across multiple levels of detail, from the broad environment down to the specific way people are positioned relative to one another.

Methods

The authors propose the Multi-Granularity Reasoning (MGR) framework, which processes an image through two main branches:

  1. Global Branch: Uses a deep CNN (ResNet-101) to extract features from the entire image, capturing "scene-level" knowledge (e.g., an office vs. a park).
  2. Regional & Interaction Branch: This is the core innovation. It uses two specialized graphs to model relationships:
  3. Person-Object Graph (POG): Connects people to the objects around them. If a person's pose keypoint (like a hand) intersects with an object (like a laptop), the graph establishes a functional connection.
  4. Person-Pose Graph (PPG): Models the physical interaction between two people based on 17 skeleton keypoints. It distinguishes between "active" nodes (hands, wrists, ankles) and "passive" nodes to capture gestures like hand-holding or walking together.

These graphs are processed by Graph Convolutional Networks (GCNs), which perform "relational reasoning" by passing messages between nodes. Finally, the scores from the global CNN and the GCNs are fused to make a final prediction.

Datasets

The framework was evaluated on two large-scale public benchmarks:

  • People in Social Context (PISC): Contains ~55,000 instances across six fine-level relations (Friend, Family, Couple, Professional, Commercial, and No Relation).
  • People in Photo Album (PIPA): A dataset based on social domain theory, featuring 16 different social relations across five domains.

Key Insights and Ideas

  • Pose as a Proxy for Action: The most significant "aha" moment is using pose keypoints to guide graph construction. Instead of just noting that a person and a bag exist in the same frame, the model checks if the person is actually interacting with the bag.
  • Context Matters Differently: The authors found that "non-intimate" relations (like Professional or Commercial) are more highly correlated with the global scene (e.g., an office setting), while "intimate" relations (like Family or Friends) rely more heavily on fine-grained interactions and poses.
  • Complementary Features: The ablation study showed that while individual modules (Global, POG, PPG) perform well, their combination is significantly more powerful, proving that social cues are distributed across different levels of granularity.

Key Results

The MGR framework outperformed previous state-of-the-art models on both major datasets:

  • PISC Dataset: Achieved a mean Average Precision (mAP) of 70.0%, a notable improvement over the previous best of 68.7% (GRM).
  • PIPA Dataset: Achieved 64.4% accuracy, surpassing the previous state-of-the-art of 62.3%.
  • Ablation Success: Adding pose guidance to the Person-Object Graph (POG) improved its mAP from 65.5% to 66.3%, validating that knowing how a person touches an object is better than just knowing they are near it.

Limitations and Future Work

Despite the improvements, the authors note that social recognition remains "very challenging." * Ambiguity: Intimate relations (friends vs. family) often look very similar and are easily confused even by humans. * Data Imbalance: Real-world datasets are often "unbalanced," with many examples of some relations and very few of others, which can cause deep learning models to struggle with rare categories. * Future Direction: The authors plan to explore more sophisticated "context cues" in the global scene to better handle these unbalanced samples.

Summary updated

Abstract

Discovering social relations in images can make machines better interpret the behavior of human beings. However, automatically recognizing social relations in images is a challenging task due to the significant gap between the domains of visual content and social relation. Existing studies separately process various features such as faces expressions, body appearance, and contextual objects, thus they cannot comprehensively capture the multi-granularity semantics, such as scenes, regional cues of persons, and interactions among persons and objects. To bridge the domain gap, we propose a Multi-Granularity Reasoning framework for social relation recognition from images. The global knowledge and mid-level details are learned from the whole scene and the regions of persons and objects, respectively. Most importantly, we explore the fine-granularity pose keypoints of persons to discover the interactions among persons and objects. Specifically, the pose-guided Person-Object Graph and Person-Pose Graph are proposed to model the actions from persons to object and the interactions between paired persons, respectively. Based on the graphs, social relation reasoning is performed by graph convolutional networks. Finally, the global features and reasoned knowledge are integrated as a comprehensive representation for social relation recognition. Extensive experiments on two public datasets show the effectiveness of the proposed framework.
BibTeX
@inproceedings{zhang2019multi,
  author = {Meng Zhang and Xinchen Liu and Wu Liu and Anfu Zhou and Huadong Ma and Tao Mei},
  title = {{Multi-Granularity Reasoning for Social Relation Recognition From Images}},
  booktitle = {ICME},
  year = {2019},
  doi = {10.1109/icme.2019.00279},
}

Loading related content...

Figure