Introduction
Image captioning has achieved significant milestones on natural image datasets like MS-COCO. However, game scene understanding remains a distinct challenge due to the high semantic complexity and action-oriented nature of video game environments. Standard "Bottom-Up and Top-Down" (BUTD) models often suffer from information loss during regional feature extraction and struggle to effectively fuse these features with global context. This paper introduces the ResTD_BU (Residual Top-Down Bottom-Up) model, which leverages multi-level residual attention to generate more descriptive and human-like captions for game scenes.
The ResTD_BU Architecture
The proposed model enhances the standard BUTD framework by introducing attention mechanisms at two critical stages of the pipeline: feature extraction and caption generation. By applying residual learning principles—similar to ResNet—the model ensures that original visual features are preserved even as the attention mechanism learns to focus on specific regions.
Multi-Level Residual Attention
The core innovation lies in two specific modules:
- Lower-Level Residual Attention (ResTD_1): Integrated directly into the bottom-up network (based on Faster R-CNN), this module replaces simple average pooling. It uses a Multi-Layer Perceptron (MLP) to learn an attention distribution over detected regions, preventing the loss of critical spatial details.
- Upper-Level Residual Attention (ResTD_2): Located within the LSTM-based caption generator, this module helps the model selectively attend to image regions while combining them with previously generated words and global image context.
Specialized Game Datasets
To address the lack of specialized benchmarks, the researchers developed two new datasets:
- GD_Det: A game object detection dataset with 7 classes (e.g., human, gun, monster) across ~7,000 images.
- GD_ImgCap: A captioning dataset featuring 7,558 images from 131 different games, with an average of two captions per image.
These datasets were supplemented with pre-training on established datasets like Visual Genome and V-COCO to improve the model's foundational understanding of actions and objects.
Experimental Results
The ResTD_BU model consistently outperformed state-of-the-art baselines on the GD_ImgCap dataset. Quantitative improvements were observed across all major metrics:
- BLEU-4: 19.59 (compared to 17.89 for the baseline TD model).
- CIDEr: 127.90 (compared to 117.77 for the baseline).
- SPICE: 32.67 (compared to 30.53 for the baseline).
Qualitatively, the model demonstrated a superior ability to identify specific actions. For instance, while a baseline might generate a generic caption like "A man standing," the ResTD_BU model correctly identified more nuanced interactions, such as "A man grabbing a boy's arm."
Conclusion
By integrating residual attention at multiple levels, the ResTD_BU model effectively mitigates information loss and improves feature fusion in game scene understanding. While the current datasets are smaller than general-purpose benchmarks, this work establishes a robust framework for future research into complex, action-heavy image captioning tasks.
Summary updated