Attention-Aligned Transformer for Image Captioning

Zhengcong FeiAAAI 2022 · DOI

Introduction

Image captioning requires models to translate visual information into natural language descriptions. Central to this task is the attention mechanism, which aims to "ground" generated words in specific image regions. However, standard Transformer architectures often suffer from "deviated focus"—a phenomenon where the regions receiving the highest attention weights are not the features most responsible for the model's output. The $A^2$ Transformer (Attention-Aligned Transformer) addresses this by aligning attention weights with the truly decisive visual features identified through self-supervised perturbation.

The Challenge of Deviated Focus

In ideal grounding, when a model generates the word "sink," its attention should be centered on the sink. Empirically, however, the correlation between attention weights and gradient-based importance metrics is weak (often below 0.2). This inconsistency leads to captions that may be fluent but are visually inaccurate. For instance, a model might attend to a "shelf" but incorrectly predict the word "sink." This gap between attention and actual influence impairs both performance and interpretability.

The $A^2$ Transformer Framework

The $A^2$ Transformer introduces a self-supervised learning approach to identify and reinforce influential features without requiring manual grounding annotations.

1. Learnable Mask Perturbation Network

This component applies a learnable mask ($m_t$) to the original attention weights. The network is trained to find the minimal perturbation that results in the maximum degradation of the captioning loss. By "poking" the model's features and observing which disturbances hurt performance most, the system identifies the regions that are truly decisive for the output.

2. Attention-Aligned Network

Once influential regions are identified, the model refines its original attention weights. The authors evaluated several fusion strategies, including Max Pooling, Moving Average, and Exponential Decay. A Gating Mechanism proved most effective, using a learnable gate to dynamically balance the original and updated weights.

The training process involves an initial cross-entropy optimization followed by self-critical reinforcement learning to maximize the CIDEr score.

Performance and Qualitative Analysis

The $A^2$ Transformer achieves state-of-the-art results on the MS COCO dataset. On the Karpathy test split, it reached a CIDEr score of 133.9, surpassing previous benchmarks like M2-T and AoANet. It also secured a top position on the official MS COCO online leaderboard with a score of 134.7.

Beyond numerical metrics, the model demonstrates superior qualitative grounding. Human evaluations indicate significant gains in "Richness," as the aligned attention allows the model to incorporate more accurate and specific descriptive details.

Comparison of attention maps between a standard Transformer (blue) and the $A^2$ Transformer (red). The standard model exhibits deviated focus, attending to a shelf while predicting the word 'sink'.

Analysis of the model's internal dynamics revealed that higher Transformer layers require more alignment than lower layers. This suggests that while lower layers capture general visual features, higher layers are more susceptible to deviated focus when mapping visuals to linguistic tokens.

Conclusion

The $A^2$ Transformer effectively bridges the gap between attention weights and feature importance. By employing a perturbation-based self-supervised network, the model ensures that its internal focus aligns with the visual evidence required for accurate captioning. This approach not only improves state-of-the-art performance but also enhances the transparency and interpretability of multi-modal Transformer models. Future work may extend this alignment strategy to other tasks such as Visual Question Answering (VQA) and video analysis.

Summary updated

BibTeX
@inproceedings{fei2022attention,
  author = {Zhengcong Fei},
  title = {{Attention-Aligned Transformer for Image Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i1.19940},
}
Figure