Multi-Gate Attention Network for Image Captioning

Weitao Jiang, Xiying Li, Haifeng Hu, Qiang Lu, Bohong LiuIEEE Access 2021 · OpenReview

Abstract

Self-attention mechanisms have become a cornerstone of the encoder-decoder framework in image captioning, facilitating enhanced feature representation and relevant information capture. However, conventional self-attention often suffers from the "everything is relevant" fallacy, assigning weights to all candidate vectors regardless of their actual contribution. Furthermore, existing methods frequently overlook intra-object noise and rely on complex NLP-centric Transformer architectures. This paper introduces the Multi-Gate Attention Network (MGAN), featuring a Multi-Gate Attention (MGA) block and a simplified pre-layernorm transformer. The MGA block incorporates an Attention Weight Gate (AWG) to filter irrelevant regions and a Self-Gated (SG) module to suppress internal feature noise. Experimental results on the MS COCO dataset demonstrate that MGAN achieves state-of-the-art performance and exhibits strong generalizability across different architectures.

Introduction

Modern image captioning relies heavily on attention mechanisms to bridge the gap between visual features and linguistic descriptions. Despite their success, standard self-attention models face significant hurdles. First, the Softmax function inherently distributes attention across all inputs, including irrelevant background noise. Second, features extracted from object detectors like Faster R-CNN often contain "leakage" from surrounding areas, yet are treated as noise-free. Finally, the standard Transformer architecture, originally designed for text, is often unnecessarily heavy for visual feature refinement.

Standard Self-Attention mechanism which the proposed MGA block aims to improve by adding gating modules.

Proposed Method: Multi-Gate Attention Network (MGAN)

The core of MGAN is the Multi-Gate Attention (MGA) block, designed to enhance selectivity at both the inter-object and intra-object levels.

Multi-Gate Attention (MGA) Block

The MGA block extends traditional self-attention with two primary components:

  1. Attention Weight Gate (AWG): This module applies a hard thresholding mechanism to attention weights. By masking out weights below a specific value (e.g., the mean or the $l$-th largest value), the model is forced to focus on the most significant visual cues.
  2. Self-Gated (SG) Module: This module addresses intra-object noise by applying a sigmoid-based gate to the feature vectors themselves. It re-weights internal dimensions to suppress irrelevant information before the attention calculation.

Pre-layernorm Transformer

To improve efficiency, the authors propose a modified Transformer encoder. By moving Layer Normalization to the start of the block and removing the feed-forward network (FFN), the architecture becomes more lightweight and stable during training, eliminating the need for a learning rate warm-up phase.

Comparison between the (a) original Transformer architecture and the proposed (b) Pre-layernorm transformer used in MGAN.

Experimental Results

The MGAN was evaluated on the MS COCO dataset using the Karpathy split. The model achieved a CIDEr score of 118.8 under cross-entropy loss, outperforming several state-of-the-art baselines.

Beyond raw performance, the MGA block demonstrated remarkable generalizability. When integrated into the standard Up-Down captioning model, it yielded a 4.2-point increase in CIDEr score. Qualitative analysis further showed that MGAN produces more precise and contextually accurate captions by effectively filtering out misleading visual information.

Conclusion

The Multi-Gate Attention Network addresses critical inefficiencies in standard attention-based image captioning. By introducing selective gating at multiple levels and simplifying the Transformer backbone, MGAN provides a more efficient and accurate framework for generating descriptive captions. Future work may explore end-to-end training and more complex geometric relationship modeling to further push the boundaries of visual-to-text translation.

Summary updated

Abstract

Self-attention mechanism, which has been successfully applied to current encoder-decoder framework of image captioning, is used to enhance the feature representation in the image encoder and capture the most relevant information for the language decoder. However, most existing methods will assign attention weights to all candidate vectors, which implicitly hypothesizes that all vectors are relevant. Moreover, current self-attention mechanisms ignore the intra-object attention distribution, and only consider the inter-object relationships. In this paper, we propose a Multi-Gate Attention (MGA) block, which expands the traditional self-attention by equipping with additional Attention Weight Gate (AWG) module and Self-Gated (SG) module. The former constrains the attention weights to be assigned to the most contributive objects. The latter is adopted to consider the intra-object attention distribution and eliminate the irrelevant information in object feature vector. Furthermore, most current image captioning methods apply the original transformer designed for natural language processing task, to refine image features directly. Therefore, we propose a pre-layernorm transformer to simplify the transformer architecture and make it more efficient for image feature enhancement. By integrating MGA block with pre-layernorm transformer architecture into the image encoder and AWG module into the language decoder, we present a novel Multi-Gate Attention Network (MGAN). The experiments on MS COCO dataset indicate that the MGAN outperforms most of the state-of-the-art, and further experiments on other methods combined with MGA blocks demonstrate the generalizability of our proposal.
BibTeX
@inproceedings{jiang2021multi,
  author = {Weitao Jiang and Xiying Li and Haifeng Hu and Qiang Lu and Bohong Liu},
  title = {{Multi-Gate Attention Network for Image Captioning}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=O75bQSaNlt},
}
Figure