Abstract
Self-attention mechanisms have become a cornerstone of the encoder-decoder framework in image captioning, facilitating enhanced feature representation and relevant information capture. However, conventional self-attention often suffers from the "everything is relevant" fallacy, assigning weights to all candidate vectors regardless of their actual contribution. Furthermore, existing methods frequently overlook intra-object noise and rely on complex NLP-centric Transformer architectures. This paper introduces the Multi-Gate Attention Network (MGAN), featuring a Multi-Gate Attention (MGA) block and a simplified pre-layernorm transformer. The MGA block incorporates an Attention Weight Gate (AWG) to filter irrelevant regions and a Self-Gated (SG) module to suppress internal feature noise. Experimental results on the MS COCO dataset demonstrate that MGAN achieves state-of-the-art performance and exhibits strong generalizability across different architectures.
Introduction
Modern image captioning relies heavily on attention mechanisms to bridge the gap between visual features and linguistic descriptions. Despite their success, standard self-attention models face significant hurdles. First, the Softmax function inherently distributes attention across all inputs, including irrelevant background noise. Second, features extracted from object detectors like Faster R-CNN often contain "leakage" from surrounding areas, yet are treated as noise-free. Finally, the standard Transformer architecture, originally designed for text, is often unnecessarily heavy for visual feature refinement.
Proposed Method: Multi-Gate Attention Network (MGAN)
The core of MGAN is the Multi-Gate Attention (MGA) block, designed to enhance selectivity at both the inter-object and intra-object levels.
Multi-Gate Attention (MGA) Block
The MGA block extends traditional self-attention with two primary components:
- Attention Weight Gate (AWG): This module applies a hard thresholding mechanism to attention weights. By masking out weights below a specific value (e.g., the mean or the $l$-th largest value), the model is forced to focus on the most significant visual cues.
- Self-Gated (SG) Module: This module addresses intra-object noise by applying a sigmoid-based gate to the feature vectors themselves. It re-weights internal dimensions to suppress irrelevant information before the attention calculation.
Pre-layernorm Transformer
To improve efficiency, the authors propose a modified Transformer encoder. By moving Layer Normalization to the start of the block and removing the feed-forward network (FFN), the architecture becomes more lightweight and stable during training, eliminating the need for a learning rate warm-up phase.
Experimental Results
The MGAN was evaluated on the MS COCO dataset using the Karpathy split. The model achieved a CIDEr score of 118.8 under cross-entropy loss, outperforming several state-of-the-art baselines.
Beyond raw performance, the MGA block demonstrated remarkable generalizability. When integrated into the standard Up-Down captioning model, it yielded a 4.2-point increase in CIDEr score. Qualitative analysis further showed that MGAN produces more precise and contextually accurate captions by effectively filtering out misleading visual information.
Conclusion
The Multi-Gate Attention Network addresses critical inefficiencies in standard attention-based image captioning. By introducing selective gating at multiple levels and simplifying the Transformer backbone, MGAN provides a more efficient and accurate framework for generating descriptive captions. Future work may explore end-to-end training and more complex geometric relationship modeling to further push the boundaries of visual-to-text translation.
Summary updated