Multimodal Graph Transformer for Multimodal Question Answering

Xuehai He, Xin Eric Wang

Introduction

In the landscape of vision-and-language tasks, Transformer-based architectures have become the dominant paradigm. These models excel at processing vast amounts of data by learning implicit relationships through self-attention mechanisms. However, a significant challenge remains: Transformers are often "implicit" learners that struggle to capture the explicit, structured relationships inherent in the physical world. For instance, while a standard Transformer might correlate the words "shirt" and "person," it may not inherently grasp the structural relationship that the shirt is on the person unless specifically guided.

Conversely, Graph Neural Networks (GNNs) are designed specifically to handle such structured data, representing entities as nodes and relationships as edges. While GNNs provide strong inductive biases for reasoning, they often lack the scalability and raw performance of large-scale Transformers. This paper introduces the Multimodal Graph Transformer, a framework designed to bridge this gap by integrating explicit structural knowledge directly into the Transformer’s attention mechanism.

Methodology

The core innovation of this work is the introduction of a "quasi-attention" mechanism that allows the model to utilize graph-based priors without sacrificing the efficiency of the Transformer architecture.

1. Multimodal Graph Construction

To provide the model with structured context, the authors construct three distinct types of graphs from the input data:

  • Text Graph: This graph extracts entities and their linguistic relationships directly from the question text, providing a roadmap for the query's logic.
  • Semantic Graph: Utilizing a scene graph parser, the model transforms sentences or tabular data into a network of entities and their specific properties (e.g., color, size, or state).
  • Dense Region Graph: The visual input is processed by slicing the image into patches. These patches are treated as nodes in a nearly fully-connected graph, allowing the model to capture spatial features and local-to-global visual context.

2. The Quasi-Attention Mechanism

Rather than treating the graph as a separate input to be processed by a GNN, the authors propose a graph-involved plug-and-play quasi-attention mechanism. This mechanism modifies the standard self-attention formula to include structural priors:

$$\text{Attention} = \text{SoftMax}\left(\frac{QK^T}{\sqrt{d}} + G + \lambda\hat{G}\right)V$$

In this formulation:

  • $G$ (Fixed Prior): An adjacency matrix derived from the constructed graphs. It acts as a structural mask, guiding the model toward features that are logically related based on prior knowledge.
  • $\hat{G}$ (Trainable Bias): A learned matrix that allows the model to refine its understanding of these relationships during the training process.
  • $\lambda$: A hyperparameter used to balance the influence of the fixed structural prior against the learned bias.

This approach effectively regularizes the attention mechanism, forcing the model to align features across different modalities—such as grounding the word "apple" in the text graph to a specific circular red region in the visual graph.

Experimental Evaluation

The Multimodal Graph Transformer was evaluated against several state-of-the-art models across three demanding benchmarks:

  1. GQA: A dataset specifically designed for real-world visual reasoning and multi-hop questions, requiring the model to navigate complex spatial and semantic relationships.
  2. VQAv2: The industry-standard benchmark for visual question answering, which includes balancing to mitigate language-only biases.
  3. MultiModalQA (MMQA): A complex task requiring joint reasoning across three distinct sources: images, text, and tables.

Results and Key Insights

The experimental results demonstrate the efficacy of injecting explicit structure into the attention process:

  • Superior Reasoning on GQA: The model achieved an 68.7% overall accuracy, significantly outperforming established baselines such as LXMERT (60.0%) and the Neural State Machine (NSM) (63.2%).
  • Multi-Source Integration: On the MultiModalQA dataset, the Graph Transformer provided a notable boost, increasing performance by approximately 2% in F1 score and 4% in Exact Match (EM) compared to standard Transformer architectures.
  • Ablation Studies: To confirm the impact of the graph components, the authors conducted ablation tests. Removing the graph-based priors resulted in a performance drop from 72.4% to 65.4% on GQA, confirming that the structured information is the primary driver of the model's reasoning capabilities.

Discussion and Conclusion

The Multimodal Graph Transformer offers a versatile, "plug-and-play" solution for enhancing reasoning in vision-and-language tasks. Because the quasi-attention mechanism modifies the attention score calculation rather than the underlying architecture, it can be integrated into various Transformer backbones, such as ViT or LXMERT.

Despite its success, the model faces challenges common to the field. It remains susceptible to the inherent biases of its pretrained backbones, and its performance is partially dependent on the quality of the initial graph construction; errors in the scene graph parser can propagate through the network. Future work will likely focus on more robust graph representation learning and extending this structured attention framework to broader multimodal reasoning tasks.

Summary updated

Abstract

Despite the success of Transformer models in vision and language tasks, they often learn knowledge from enormous data implicitly and cannot utilize structured input data directly. On the other hand, structured learning approaches such as graph neural networks (GNNs) that integrate prior information can barely compete with Transformer models. In this work, we aim to benefit from both worlds and propose a novel Multimodal Graph Transformer for question answering tasks that requires performing reasoning across multiple modalities. We introduce a graph-involved plug-and-play quasi-attention mechanism to incorporate multimodal graph information, acquired from text and visual data, to the vanilla self-attention as effective prior. In particular, we construct the text graph, dense region graph, and semantic graph to generate adjacency matrices, and then compose them with input vision and language features to perform downstream reasoning. Such a way of regularizing self-attention with graph information significantly improves the inferring ability and helps align features from different modalities. We validate the effectiveness of Multimodal Graph Transformer over its Transformer baselines on GQA, VQAv2, and MultiModalQA datasets.
BibTeX
@inproceedings{he2023multimodal,
  author = {Xuehai He and Xin Eric Wang},
  title = {{Multimodal Graph Transformer for Multimodal Question Answering}},
  booktitle = {EACL},
  year = {2023},
  url = {https://openreview.net/forum?id=8dkAsEvq0C},
}
Figure