Causal Event Graph-Guided Language-based Spatiotemporal Question Answering

Kaushik Roy, Alessandro Oltramari, Yuxin Zi, Chathurangi Shyalika, Vignesh Narayanan, Amit P. ShethAAAI 2024 · DOI

Introduction

Presented at the AAAI Spring Symposium (SSS-24), this research explores moving Large Language Models (LLMs) beyond simple pattern matching toward a robust "world model" capable of understanding cause and effect. The core innovation involves enhancing spatiotemporal reasoning by requiring models to predict Causal Event Graphs (CEGs) alongside answers to video-based questions.

The Challenge: Intuitive Physics in LLMs

While LLMs excel at language patterns, they often lack "intuitive physics." For instance, when observing billiard balls colliding, an LLM might predict subsequent movement based on linguistic probability rather than a physical understanding of momentum. Existing Spatiotemporal Question Answering (QA) methods typically rely on black-box pattern recognition or neurosymbolic methods that generate functional programs. However, these do not necessarily prove an underlying grasp of physical causality.

Methodology: Causal Event Graphs as Proof

The proposed framework treats spatiotemporal QA as a dual task: answering the question (or generating a functional program) and reconstructing a Causal Event Graph (CEG).

Illustration of the spatiotemporal QA process, including the functional program and the corresponding Causal Event Graph (CEG).
  • CEG Structure: Nodes represent discrete events (e.g., "red ball hits green cube"), while edges denote causal relationships (e.g., "Event A caused Event B").
  • Model Architecture: An autoregressive model $M(Q, \theta)$ predicts tokens, while a graph reconstruction function $G(Q, \theta')$ predicts the CEG's adjacency matrix using node embeddings.
  • Synergistic Loss Function: To balance these tasks, the authors introduced a question-specific Lagrange multiplier network. This two-layer neural network dynamically weights the graph-reconstruction loss against the text-prediction loss, ensuring mutual reinforcement.
  • Theoretical Foundation: The authors provide a formal proof of convergence using Lyapunov functions, ensuring the stability of the combined Gradient Descent and Lagrange multiplier approach.

Benchmarking on Synthetic Dynamics

The study utilizes two primary benchmarks featuring synthetic videos of colliding objects:

Sample frames from the CLEVRER dataset showing dynamic object interactions over time.
  • CLEVRER: Provides videos, QA pairs, and ground-truth functional programs. CEGs were extracted from the dataset's rich metadata.
  • CLEVRER-Humans: Features natural language questions and human-curated CEGs, offering more complex and less "perfect" data than the synthetic version.

Key Results and Insights

  • High Performance: The method achieved 91.85% accuracy across combined datasets. While slightly below the absolute leaderboard peak (95.24%), it offers superior interpretability.
  • Graph Fidelity: The model reconstructed causal graphs with ≥ 98.3% accuracy, demonstrating a strong "mental model" of the scene.
  • Intrinsic Knowledge Proof: The predicted CEG serves as a proof of reasoning. Correctly identifying causal chains before answering a related question increases confidence in the model's logic.
  • Custom Embedding Success: Standard graph embedding methods (like TransE) performed poorly (hits@1 < 0.6), justifying the authors' custom reconstruction method based on KL divergence.

Conclusion and Future Directions

The current framework demonstrates that forcing LLMs to "show their work" through causal graphs significantly enhances reasoning. Future work will aim to move beyond the "closed world" of CLEVRER into real-world video datasets—such as traffic accidents or complex human actions—where causal relationships are noisier and more ambiguous.

Summary updated

BibTeX
@inproceedings{roy2024causal,
  author = {Kaushik Roy and Alessandro Oltramari and Yuxin Zi and Chathurangi Shyalika and Vignesh Narayanan and Amit P. Sheth},
  title = {{Causal Event Graph-Guided Language-based Spatiotemporal Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v3i1.31204},
}
Figure