Multimodal Event Causality Reasoning with Scene Graph Enhanced Interaction Network

Jintao Liu, Kaiwen Wei, Chenglong LiuAAAI 2024 · DOI

Abstract

Understanding the causal relationships between events is a fundamental aspect of human intelligence. In the digital domain, Multimodal Event Causality Reasoning (MECR) aims to infer these relationships from combined textual and visual data. We introduce the Scene Graph Enhanced Interaction Network (SEIN), a novel framework that utilizes structured scene graphs to capture intra-image object relationships and Optimal Transport to align entities across consecutive frames. Our approach significantly improves the model's ability to reason about event progression, achieving state-of-the-art results on the Vis-Causal dataset.

Introduction: The Challenge of Causal Reasoning

Causality reasoning involves identifying the link between a cause and its subsequent effect. In multimodal contexts, this requires a model to not only recognize objects but also understand their interactions and how they evolve over time. For instance, identifying that a dog's jump is caused by a girl throwing a plate requires tracking the plate's trajectory and the dog's reaction across images.

Existing MECR models often fall short in two key areas:

  1. Intra-image Context: They frequently fail to capture the relational context between objects within a single frame (e.g., a "girl" is "holding" a "plate").
  2. Inter-image Dynamics: They struggle to effectively track object transformations or movements between consecutive images, which is essential for perceiving event flow.

The SEIN Architecture

To address these limitations, the Scene Graph Enhanced Interaction Network (SEIN) employs a multi-stage pipeline designed to extract and align structured information from multimodal inputs.

The SEIN architecture integrates Scene Graph Generation, Graph Convolutional Networks, and Optimal Transport for robust causal reasoning.

The architecture consists of four primary components:

  • Scene Graph Generation (SGG): Utilizing Faster R-CNN and specialized toolkits, SEIN transforms raw pixels into structured graphs where nodes represent objects and edges represent their semantic relationships (e.g., near, holding, on).
  • Graph Convolutional Networks (GCN): These networks process the generated scene graphs to produce context-aware embeddings. This ensures that an object's representation is influenced by its surrounding environment.
  • Optimal Transport (OT) Alignment: To bridge the temporal gap between two images, SEIN uses an Optimal Transport-based strategy. This mathematical framework identifies the most efficient mapping of objects from the first frame to the second, allowing the model to recognize visual changes and object persistence over time.
  • Adaptive Prediction Strategy: Recognizing that causal cues can vary in modality, SEIN calculates confidence scores for both textual and multimodal branches. An adaptive weight factor ($\beta$) dynamically balances these branches to produce the final prediction.

Experimental Evaluation

The model was rigorously evaluated on the Vis-Causal dataset, which comprises 1,609 image pairs and over 82,000 candidate events derived from YouTube videos of daily activities. The task requires the model to select the correct "effect" event from a set of candidates given an initial "cause" event.

Key Results

SEIN established new state-of-the-art (SOTA) benchmarks on the Vis-Causal dataset, notably outperforming previous leading models such as OARNet and iReason.

Quantitative comparison showing SEIN's superior performance in R@1, R@5, and R@10 metrics compared to baseline models.

As shown in the results, SEIN achieved an R@1 of 18.09% and an R@10 of 73.40%. Ablation studies further revealed that the Scene Graph component is the most critical element of the architecture; its removal led to the most significant performance degradation. The model showed particular strength in categories like "Socializing" and "Care," where human-object interactions are dense and complex.

Conclusion and Future Directions

SEIN demonstrates that structured, graph-based representations are vital for moving beyond simple pattern matching toward logical causal understanding. By explicitly modeling object relationships and temporal alignment, SEIN provides a more robust framework for multimodal reasoning.

While SEIN is a significant step forward, its performance remains dependent on the quality of initial Scene Graph Generation. Future work will focus on reducing this dependency and incorporating broader external knowledge bases to enhance the model's "common sense" reasoning in scenarios where visual evidence is ambiguous.

Summary updated

BibTeX
@inproceedings{liu2024multimodal,
  author = {Jintao Liu and Kaiwen Wei and Chenglong Liu},
  title = {{Multimodal Event Causality Reasoning with Scene Graph Enhanced Interaction Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i8.28724},
}
Figure