Scene-based Factored Attention for Image Captioning

Chen Shen, Rongrong Ji, Fuhai Chen, Xiaoshuai Sun, Xiangming LiarXiv 2019 · OpenReview

Abstract

Image captioning has attracted ever-increasing research attention in the multimedia community. To this end, most cutting-edge works rely on an encoder-decoder framework with attention mechanisms, which have achieved remarkable progress. However, such a framework does not consider scene concepts to attend visual information, which leads to sentence bias in caption generation and defects the performance correspondingly. We argue that such scene concepts capture higher-level visual semantics and serve as an important cue in describing images. In this paper, we propose a novel scene-based factored attention module for image captioning. Specifically, the proposed module first embeds the scene concepts into factored weights explicitly and attends the visual information extracted from the input image. Then, an adaptive LSTM is used to generate captions for specific scene types. Experimental results on Microsoft COCO benchmark show that the proposed scene-based attention module improves model performance a lot, which outperforms the state-of-the-art approaches under various evaluation metrics.

Introduction

Modern image captioning models typically utilize an encoder-decoder framework with attention mechanisms to focus on specific image regions or detected objects. While these "what" and "where" cues are powerful, they are often insufficient for precise word selection. For instance, a person lying on snow is likely "laying" (perhaps after a fall), whereas a person in a bed is likely "sleeping."

Existing models frequently suffer from sentence bias, relying too heavily on common word patterns in training data rather than the specific environmental context of the image. This paper introduces a scene-based factored attention module that explicitly incorporates high-level scene context (e.g., "outdoor" vs. "indoor") to resolve such ambiguities. By factorizing attention weights based on scene concepts, the model provides a "macro" context to guide the interpretation of "micro" details like objects and regions.

Figure 1: Motivation showing how scene context affects word probability distributions.

Methodology

The proposed architecture integrates hierarchical visual information through three distinct levels:

  1. Low-level Features: Regional features extracted using a Faster R-CNN.
  2. Mid-level Concepts: Object-centered visual concepts (e.g., "person," "bed") from a dedicated CNN.
  3. High-level Scene Concepts: Environment-level context (e.g., "bedroom," "ocean") extracted via a CNN pretrained on the Places dataset.

Factored Attention Module

The core innovation is the factored attention module. Rather than simple concatenation, scene concepts are embedded into a diagonal matrix $S$, which factorizes the weight matrix $W_h$ of the attention mechanism ($W_h = U_h S V_h$). This allows the scene context to act as a dynamic filter, explicitly re-weighting the importance of regional and object features based on the overall environment.

Two-Layer LSTM

The model employs a two-layer LSTM structure. The first LSTM generates a hidden state that, when combined with scene concepts, drives the factored attention. The resulting attentive visual information is then processed by a second LSTM to generate the final caption.

Experimental Evaluation

The model was evaluated on the Microsoft COCO (MS COCO) dataset using the standard Karpathy split (113,287 training, 5K validation, and 5K test images).

Key Results

The proposed model achieved state-of-the-art results on the MS COCO benchmark:

  • Quantitative Performance: On the Karpathy test split, the model reached a CIDEr score of 1.147 with standard MLE training and 1.268 with reinforcement learning (RL)-based training. This outperforms previous leading models such as Up-Down and DA across BLEU, METEOR, ROUGE, and SPICE metrics.
  • Qualitative Analysis: Attention heatmaps demonstrate that the scene-based model focuses more discriminatively on relevant regions. For example, in a skiing scene, the model correctly identifies "laying" instead of "sleeping" by incorporating the outdoor context.

Insights

  • Scene as a Tie-Breaker: Scene concepts effectively resolve ambiguity between visually similar actions by providing environmental context.
  • Factorization vs. Concatenation: Explicitly factorizing attention weights proved superior to simple concatenation, allowing for more direct interaction between high-level scenes and lower-level visual features.
  • Hierarchical Guidance: The model mimics a "top-down" human-like approach—first identifying the environment, then focusing on relevant details.

Conclusion

While the scene-based factored attention module significantly improves captioning accuracy, its performance is currently dependent on the accuracy of the underlying scene classifier. Future work may explore more open-ended or learned scene representations and specialized reinforcement learning objectives to further refine the use of scene-dependent keywords.

Summary updated

Abstract

Image captioning has attracted ever-increasing research attention in the multimedia community. To this end, most cutting-edge works rely on an encoder-decoder framework with attention mechanisms, which have achieved remarkable progress. However, such a framework does not consider scene concepts to attend visual information, which leads to sentence bias in caption generation and defects the performance correspondingly. We argue that such scene concepts capture higher-level visual semantics and serve as an important cue in describing images. In this paper, we propose a novel scene-based factored attention module for image captioning. Specifically, the proposed module first embeds the scene concepts into factored weights explicitly and attends the visual information extracted from the input image. Then, an adaptive LSTM is used to generate captions for specific scene types. Experimental results on Microsoft COCO benchmark show that the proposed scene-based attention module improves model performance a lot, which outperforms the state-of-the-art approaches under various evaluation metrics.
BibTeX
@misc{shen2019scene,
  author = {Chen Shen and Rongrong Ji and Fuhai Chen and Xiaoshuai Sun and Xiangming Li},
  title = {{Scene-based Factored Attention for Image Captioning}},
  year = {2019},
  eprint = {1908.02632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.02632},
}
Figure