Image Captioning with Integrated Bottom-Up and Multi-level Residual Top-Down Attention for Game Scene Understanding

Jian Zheng, Sudha Krishnamurthy, Ruxin Chen, Min-Hung Chen, Zhenhao Ge, Xiaohua LiarXiv 2019 · arXiv

Introduction

Image captioning has achieved significant milestones on natural image datasets like MS-COCO. However, game scene understanding remains a distinct challenge due to the high semantic complexity and action-oriented nature of video game environments. Standard "Bottom-Up and Top-Down" (BUTD) models often suffer from information loss during regional feature extraction and struggle to effectively fuse these features with global context. This paper introduces the ResTD_BU (Residual Top-Down Bottom-Up) model, which leverages multi-level residual attention to generate more descriptive and human-like captions for game scenes.

The ResTD_BU Architecture

The proposed model enhances the standard BUTD framework by introducing attention mechanisms at two critical stages of the pipeline: feature extraction and caption generation. By applying residual learning principles—similar to ResNet—the model ensures that original visual features are preserved even as the attention mechanism learns to focus on specific regions.

Overview of the ResTD_BU image captioning pipeline.

Multi-Level Residual Attention

The core innovation lies in two specific modules:

  1. Lower-Level Residual Attention (ResTD_1): Integrated directly into the bottom-up network (based on Faster R-CNN), this module replaces simple average pooling. It uses a Multi-Layer Perceptron (MLP) to learn an attention distribution over detected regions, preventing the loss of critical spatial details.
  2. Upper-Level Residual Attention (ResTD_2): Located within the LSTM-based caption generator, this module helps the model selectively attend to image regions while combining them with previously generated words and global image context.
The ResTD_BU Network architecture showing the integration of ResTD_1.

Specialized Game Datasets

To address the lack of specialized benchmarks, the researchers developed two new datasets:

  • GD_Det: A game object detection dataset with 7 classes (e.g., human, gun, monster) across ~7,000 images.
  • GD_ImgCap: A captioning dataset featuring 7,558 images from 131 different games, with an average of two captions per image.

These datasets were supplemented with pre-training on established datasets like Visual Genome and V-COCO to improve the model's foundational understanding of actions and objects.

Experimental Results

The ResTD_BU model consistently outperformed state-of-the-art baselines on the GD_ImgCap dataset. Quantitative improvements were observed across all major metrics:

  • BLEU-4: 19.59 (compared to 17.89 for the baseline TD model).
  • CIDEr: 127.90 (compared to 117.77 for the baseline).
  • SPICE: 32.67 (compared to 30.53 for the baseline).

Qualitatively, the model demonstrated a superior ability to identify specific actions. For instance, while a baseline might generate a generic caption like "A man standing," the ResTD_BU model correctly identified more nuanced interactions, such as "A man grabbing a boy's arm."

Conclusion

By integrating residual attention at multiple levels, the ResTD_BU model effectively mitigates information loss and improves feature fusion in game scene understanding. While the current datasets are smaller than general-purpose benchmarks, this work establishes a robust framework for future research into complex, action-heavy image captioning tasks.

Summary updated

Abstract

Image captioning has attracted considerable attention in recent years. However, little work has been done for game image captioning which has some unique characteristics and requirements. In this work we propose a novel game image captioning model which integrates bottom-up attention with a new multi-level residual top-down attention mechanism. Firstly, a lower-level residual top-down attention network is added to the Faster R-CNN based bottom-up attention network to address the problem that the latter may lose important spatial information when extracting regional features. Secondly, an upper-level residual top-down attention network is implemented in the caption generation network to better fuse the extracted regional features for subsequent caption prediction. We create two game datasets to evaluate the proposed model. Extensive experiments show that our proposed model outperforms existing baseline models.
BibTeX
@misc{zheng2019image,
  author = {Jian Zheng and Sudha Krishnamurthy and Ruxin Chen and Min-Hung Chen and Zhenhao Ge and Xiaohua Li},
  title = {{Image Captioning with Integrated Bottom-Up and Multi-level Residual Top-Down Attention for Game Scene Understanding}},
  year = {2019},
  eprint = {1906.06632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.06632},
}
Figure