Synergizing Multimodal Temporal Knowledge Graphs and Large Language Models for Social Relation Recognition

Haorui Wang, Zheng Wang, Yuxuan Zhang, Bo Wang, Bin WuEMNLP 2025

TL;DR

This paper introduces mtKG-LLM, a novel framework that combines multimodal temporal knowledge graphs (KGs) with large language models (LLMs) to improve social relation recognition in videos. By dynamically building and updating KGs from video content, mtKG-LLM provides LLMs with rich, structured temporal context, leading to state-of-the-art performance and a more robust understanding of complex social dynamics.

What Problem Does This Paper Address?

Recognizing social relations between characters in videos is a crucial but challenging task in multimodal understanding. Existing methods often struggle with the inherent graph-like structure of social networks and the temporal evolution of relationships. While Large Language Models (LLMs) have shown promise, their sequential processing nature limits their ability to effectively model complex, non-sequential graph structures. Furthermore, previous approaches often rely on coarse multimodal embeddings or struggle to capture long-range temporal dependencies, leading to limitations in accurately identifying nuanced social dynamics. This paper aims to bridge this gap by synergizing the strengths of structured knowledge representation (KGs) with the powerful reasoning capabilities of LLMs, specifically addressing the temporal and graph-structured nature of social relations in videos.

Methods

The proposed mtKG-LLM framework consists of several key components:

  • Multimodal Knowledge Graph Extraction: For each video scene, multimodal information (visual and linguistic) is extracted using a Multimodal Large Language Model (MLLM). This information is used to construct a spatial Knowledge Graph (KG) for that scene, representing characters, their interactions, and background elements.
  • Knowledge Graph Temporal Update: Temporal KGs are built by sequentially updating the spatial KGs. This process incorporates information from previous temporal KGs (long-term memory) and the current spatial KG (short-term memory), allowing for continuous temporal reasoning of social relations over time. LLMs are used to summarize and update temporal entities.
  • Community Summary: To capture global context and structural information, the temporal KGs are partitioned into communities. LLMs then generate summaries for each community, preserving structure-related information and providing a higher-level contextual understanding.
  • Social Relation Recognition: Finally, LLMs are used for social relation recognition. They leverage multi-scale information retrieved from the temporal KGs (individual, interaction, and community-level context) without fine-tuning, promoting a low-coupling and effective synergy.

Datasets

The framework is evaluated on four video datasets:

  • MovieGraphs: Used for pair-wise scene-level relation queries.
  • HLVU: Used for pair-wise scene-level relation queries.
  • ViSR: Used for video-level major relation recognition.
  • LVU: Used for video-level major relation recognition.

The paper notes that it avoids using external validated knowledge to enhance generalizability to diverse videos.

Key Insights and Ideas

  • Synergy of KGs and LLMs: The core idea is that combining structured, temporal knowledge graphs with the reasoning power of LLMs can overcome the limitations of each individual approach for complex tasks like social relation recognition.
  • Temporal Dynamics are Crucial: Social relationships are not static. The framework's ability to build and update temporal KGs is key to capturing the evolution of these relationships over time.
  • Multi-scale Context Matters: By summarizing information at both the individual/interaction level and the community level, the framework provides LLMs with a richer, multi-faceted understanding of the social context.
  • Low-Coupling Design: The framework is designed for low-coupling, allowing for easy switching between different LLMs and MLLMs, demonstrating versatility and adaptability.
  • Unsupervised KG Construction: The framework autonomously extracts information to build KGs, making it applicable to general videos without relying on pre-existing external knowledge bases.

Key Results

  • State-of-the-Art Performance: The mtKG-LLM framework achieved state-of-the-art (SOTA) performance on major datasets, with substantial improvements. For example, on the MovieGraphs dataset, the overall mean accuracy increased by 15% over previous SOTA methods.
  • Robustness Across Relations: The method demonstrates strong performance across various social relation types, particularly excelling in capturing nuanced relationships due to its ability to leverage temporal and community-level context.
  • Effectiveness of Modules: Ablation studies showed that each component (spatial KG construction, temporal KG update, and community summary) significantly contributes to the overall accuracy.
  • LLM Versatility: The framework shows encouraging performance even with smaller, open-source LLMs, though larger models like GPT-4 generally yield better results.

Limitations and Future Work

  • Low-Coupling Limits Depth: The non-invasive, low-coupling approach, while beneficial for flexibility, might limit the depth of collaboration between KGs and LLMs. Future work could explore strategies to optimize LLMs for more efficient KG processing.
  • API Costs and Latency: Using LLMs via APIs introduces network latency and ongoing usage expenses, especially for long videos. While the framework offers flexibility, optimizing for resource-constrained environments or exploring fine-tuning/distillation techniques for smaller LLMs is a direction for future research.
  • Bias in LLMs: The reliance on LLMs means the framework is susceptible to biases present in their training data, potentially affecting accuracy. Bias detection and debiasing algorithms are suggested as mitigation strategies.
  • Generalization to Specific Domains: While the framework is designed for general videos, its performance might vary on highly specialized domains without domain-specific knowledge.

Summary updated

BibTeX
@inproceedings{wang2025synergizing,
  author = {Haorui Wang and Zheng Wang and Yuxuan Zhang and Bo Wang and Bin Wu},
  title = {{Synergizing Multimodal Temporal Knowledge Graphs and Large Language Models for Social Relation Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure