Global Fusion Attention for Vision and Language Understanding (Student Abstract)

Zixin Guo, Chen Liang, Ziyu Wan, Yang Bai

Abstract

This paper introduces Global Fusion Attention (GFA), a novel mechanism for Transformer-based models designed to enhance Vision and Language (V&L) understanding. By dynamically weighting and fusing features from multiple decoder layers, GFA addresses the "coarse interaction" problem common in standard multi-layer architectures. Evaluated on the CLEVR and BabyAI benchmarks, the proposed model achieves state-of-the-art performance without requiring expensive program annotations. Notably, GFA matches the accuracy of complex baselines while utilizing significantly fewer parameters, demonstrating high efficiency in both reasoning and reinforcement learning contexts.

Introduction

Vision and Language tasks, such as Visual Question Answering (VQA), necessitate a sophisticated understanding of structured relationships between visual pixels and textual tokens. While Transformer-based decoders are standard for these tasks, they often struggle with maintaining a global perspective of information processing across layers. In a typical multi-layer decoder, intermediate representations may lose alignment with the original query, leading to erroneous outputs. This research addresses this gap by refining how multi-level features are integrated, ensuring that the model retains access to various levels of abstraction throughout the reasoning process.

Methodology

The core of this work is the Global Fusion Attention (GFA) module. Unlike traditional models that rely primarily on the final decoder layer's output, GFA treats the outputs of all decoder layers as a collective pool of information to be fused dynamically.

Architecture of the Global Fusion Attention (GFA) module.

The GFA process consists of three primary stages:

  1. Cross-Attention (CA): The model employs standard cross-attention to determine the similarity between visual features (from an encoder) and textual features (from the decoders).
  2. Learnable Weighting: For each decoder layer $i$, a specific weight $\alpha_i$ is calculated. This is achieved by concatenating the cross-attention output with the layer's internal features, followed by a linear transformation and a sigmoid function.
  3. Weighted Summation: The final representation is a weighted sum of all decoder layer outputs. This allows the model to dynamically prioritize different levels of abstraction—from low-level structural details to high-level semantics—based on the specific requirements of the question.

Experimental Setup

The authors evaluated GFA on two distinct benchmarks:

  • CLEVR: A diagnostic VQA dataset consisting of 3D-rendered objects, designed to test complex reasoning capabilities such as counting, comparison, and logical inference.
  • BabyAI: A platform for grounded language learning where agents must follow instructions in a grid-world environment, used here to assess sample efficiency in reinforcement learning.

Results and Discussion

On the CLEVR validation split, the GFA-equipped model achieved an overall accuracy of 98.9%. This performance is particularly significant because it matches or exceeds models like IEP and TbD that require 700k "program" annotations (step-by-step logical instructions) for training. GFA achieves these results using only raw images and questions.

Performance comparison on the CLEVR dataset.

As shown in the performance table, GFA demonstrates superior reasoning in sub-tasks such as "Compare Number" (99.4%) and "Count" (99.0%). Furthermore, the model is highly parameter-efficient; it achieves results comparable to the MAC (Memory, Attention, and Composition) model while using only 4.12M parameters, roughly one-third of MAC's 12.20M. In the BabyAI environment, the authors also observed improved sample efficiency, indicating that the agent requires fewer environment interactions to master instruction-following tasks.

Conclusion

Global Fusion Attention provides a robust and efficient method for integrating multi-level features in V&L tasks. By eliminating the need for program supervision and reducing parameter overhead, GFA offers a scalable path for complex reasoning models. Future work will focus on scaling this architecture to "in-the-wild" datasets like VQA v2.0 and exploring its application in more complex embodied AI environments where real-world ambiguity and physical interaction are prevalent.

Summary updated

Abstract

We extend the popular transformer architecture to a multi-modal model, processing both visual and textual inputs. We propose a new attention mechanism on Transformer-based architecture for the joint vision and language understanding tasks. Our model fuses multi-level comprehension between images and texts in a weighted manner, which could better curve the internal relationships. Experiments on benchmark VQA dataset CLEVR demonstrate the effectiveness of the proposed attention mechanism. We also observe the improvements in sample efficiency of reinforcement learning through the experiments on grounded language understanding tasks of BabyAI platform.
BibTeX
@inproceedings{guo2021global,
  author = {Zixin Guo and Chen Liang and Ziyu Wan and Yang Bai},
  title = {{Global Fusion Attention for Vision and Language Understanding (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17891},
}
Figure