FL-MSRE: A Few-Shot Learning based Approach to Multimodal Social Relation Extraction

Hai Wan, Manrong Zhang, Jianfeng Du, Ziling Huang, Yufei Yang, Jeff Z. Pan

Introduction: The Challenge of Social Relation Extraction

Social Relation Extraction (SRE) is the task of identifying the relationship between individuals mentioned in text or depicted in images. While humans naturally integrate contextual and visual cues, most AI models are unimodal, relying solely on one data type. This leads to significant hurdles in accuracy and generalizability.

Ambiguity and the Long Tail Problem

  1. Ambiguity: Textual descriptions are often vague. A sentence mentioning a "hug" could imply a husband and wife or a father and daughter. Without visual data to assess age gaps or gender, models are often forced to guess.
  2. Data Scarcity: Real-world social relations follow a "long tail" distribution. While common relations like "friend" are well-represented, rare relations like "sworn brother" suffer from extreme data scarcity, making standard deep learning models ineffective.
Visual ambiguity in social relations. A "hug" can represent various relationships depending on the individuals' attributes like age and gender.

Methodology: The FL-MSRE Framework

The authors propose FL-MSRE (Few-Shot Learning based Multimodal Social Relation Extraction), a framework that combines BERT-based text analysis with FaceNet-based facial recognition.

Multimodal Architecture

  • Sentence Encoder: Uses a pre-trained BERT model to extract textual features. Character names are masked to prevent the model from simply memorizing specific identities.
  • Face Encoder: Employs FaceNet (built on GoogLeNet) to extract features from facial bounding boxes, capturing critical attributes such as age, gender, and expression.
  • Cross-modality Encoder: Integrates text and visual features through concatenation and layer normalization to create a unified representation of the relationship.

Prototypical Networks

To handle the "long tail" problem, FL-MSRE uses Prototypical Networks. Instead of a standard classifier, the model computes a "prototype" (a mean vector) for each relation class based on a few support examples. New queries are classified by finding the nearest prototype in the embedding space using Euclidean distance.

Key Insights: Visual Cues as Error Correctors

The most significant finding is how visual data acts as a "sanity check" for text. If a text mentions a "brother" but the images show a man and a woman, the multimodal model can override the text-based guess to predict "wife" or "sister."

Multimodal disambiguation logic. Combining textual actions with visual attributes like gender and age allows for precise relation extraction.

Additionally, the researchers found that extracting faces from two entirely different images (e.g., separate headshots) performed as well as using a single photo of the pair together. This makes the model highly practical for real-world applications where group photos are rare.

Results and Performance

FL-MSRE was tested against a strong text-only BERT baseline across three new benchmarks (FC-TF, DRC-TF, and OM-TF) derived from Chinese classical literature and TV adaptations.

  • Superior Accuracy: FL-MSRE consistently outperformed the baseline, achieving an absolute improvement of 3.2% to 5.2% on the FC-TF dataset.
  • Massive Gains on Small Data: On the smaller DRC-TF dataset (3-way 1-shot), the multimodal approach improved accuracy by a staggering 22% over text-only models.
  • Stability: The multimodal model showed much lower variance, indicating more stable and reliable predictions.

Conclusion and Future Work

FL-MSRE demonstrates that visual constraints are essential for disambiguating complex human connections, especially in few-shot scenarios. Future improvements could include more sophisticated attention mechanisms for data fusion and expanding the datasets to include more diverse cultural contexts.

Summary updated

Abstract

Social relation extraction (SRE for short), which aims to infer the social relation between two people in daily life, has been demonstrated to be of great value in reality. Existing methods for SRE consider extracting social relation only from unimodal information such as text or image, ignoring the high coupling of multimodal information. Moreover, previous studies overlook the serious unbalance distribution on social relations. To address these issues, this paper proposes FL-MSRE, a few-shot learning based approach to extracting social relations from both texts and face images. Considering the lack of multimodal social relation datasets, this paper also presents three multimodal datasets annotated from four classical masterpieces and corresponding TV series. Inspired by the success of BERT, we propose a strong BERT based baseline to extract social relation from text only. FL-MSRE is empirically shown to outperform the baseline significantly. This demonstrates that using face images benefits text-based SRE. Further experiments also show that using two faces from different images achieves similar performance as from the same image. This means that FL-MSRE is suitable for a wide range of SRE applications where the faces of two people can only be collected from different images.
BibTeX
@inproceedings{wan2021fl,
  author = {Hai Wan and Manrong Zhang and Jianfeng Du and Ziling Huang and Yufei Yang and Jeff Z. Pan},
  title = {{FL-MSRE: A Few-Shot Learning based Approach to Multimodal Social Relation Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17639},
}
Figure