Reasoning with Heterogeneous Graph Alignment for Video Question Answering

Pin Jiang, Yahong Han

Introduction

Video Question Answering (VideoQA) represents a significant advancement over Image QA, requiring models to perform complex reasoning across both spatial and temporal dimensions. To answer a query such as "What does the woman do before putting her hand in her hair?", a system must simultaneously decode linguistic intent, localize specific visual actions within a sequence, and perform temporal sequencing.

Traditional approaches often rely on "late fusion," where video and text features are processed independently and combined only at the final decision stage. This often fails to capture the fine-grained, cross-modal correlations necessary for nuanced reasoning. To address this, researchers from Tianjin University proposed the Heterogeneous Graph Alignment (HGA) network, which integrates video shots and question words into a unified graph structure to facilitate joint reasoning.

The Heterogeneous Graph Alignment (HGA) Network

The HGA framework is designed to bridge the "semantic gap" between visual pixels and linguistic tokens by treating them as equal entities within a shared space. The architecture follows a structured four-stage pipeline:

  1. Feature Representation: Contextual features are extracted using Gated Recurrent Units (GRUs). Video sequences are partitioned into "shots" and processed via 2D and 3D Convolutional Neural Networks, while questions are represented using GloVe word embeddings.
  2. Modular Co-Attention Embedding (CAEO): This stage acts as a differentiable search mechanism, projecting words into the visual feature space and vice versa. This alignment ensures that the subsequent graph nodes share a common semantic grounding.
  3. Heterogeneous Graph Construction: The core innovation lies in building a graph where every video shot and every question word serves as a node. Edges are established between all nodes, with weights learned based on their semantic similarity, allowing for both intra-modality (e.g., word-to-word) and inter-modality (e.g., word-to-video) interactions.
  4. Relational Reasoning: A Graph Convolutional Network (GCN) performs message passing across the heterogeneous graph. This allows the representation of a word node to be updated by its relevant video neighbors, effectively performing cross-modal reasoning in a single pass.
Conceptual overview of the HGA framework, illustrating the alignment of video shots and question words within a heterogeneous graph.

Finally, the model fuses these "local" graph-based insights with "global" video and question summaries using bilinear fusion to generate the final answer.

Experimental Evaluation

The HGA network was rigorously tested on three benchmark VideoQA datasets:

  • TGIF-QA: A large-scale dataset focusing on actions, state transitions, and repetition counting in GIFs. HGA outperformed previous state-of-the-art (SOTA) models by up to 3.1% in counting tasks.
  • MSVD-QA: Comprising approximately 50,000 Q&A pairs, HGA achieved an accuracy of 34.7%, a 3.0% improvement over prior methods.
  • MSRVTT-QA: On this larger dataset (~243k pairs), HGA reached 35.5% accuracy, surpassing previous benchmarks by a substantial 7.6%.

Key Insights and Discussion

The success of HGA stems from its symmetrical treatment of modalities. By viewing words and shots as equivalent nodes, the model enables a more flexible and bidirectional flow of information. A critical finding was the synergy between the Co-Attention mechanism and the GCN; the former provides a clean interaction space that significantly enhances the effectiveness of the latter's relational reasoning.

However, the authors noted a specific limitation: the GCN-based reasoning occasionally introduced noise in "FrameQA" tasks—questions that can be answered using a single static image. This suggests that while complex graph reasoning is powerful for temporal tasks, it may be over-engineered for simpler, non-temporal queries. Future work will likely focus on mitigating this noise and extending the framework to handle longer-form video content where temporal dependencies are even more complex.

Summary updated

Abstract

The dominant video question answering methods are based on fine-grained representation or model-specific attention mechanism. They usually process video and question separately, then feed the representations of different modalities into following late fusion networks. Although these methods use information of one modality to boost the other, they neglect to integrate correlations of both inter- and intra-modality in an uniform module. We propose a deep heterogeneous graph alignment network over the video shots and question words. Furthermore, we explore the network architecture from four steps: representation, fusion, alignment, and reasoning. Within our network, the inter- and intra-modality information can be aligned and interacted simultaneously over the heterogeneous graph and used for cross-modal reasoning. We evaluate our method on three benchmark datasets and conduct extensive ablation study to the effectiveness of the network architecture. Experiments show the network to be superior in quality.
BibTeX
@inproceedings{jiang2020reasoning,
  author = {Pin Jiang and Yahong Han},
  title = {{Reasoning with Heterogeneous Graph Alignment for Video Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6767},
}
Figure