Application of Dual Attention Mechanism in Chinese Image Captioning

Yong Zhang, Jing ZhangJournal of Intelligent Learning Systems and Applications 2020 · DOI

Abstract

Objective: The Chinese description of images combines computer vision and natural language processing, representing a multi-mode and cross-domain challenge for artificial intelligence. This study aims to develop an image captioning model that generates natural Chinese descriptions, identifying key characters, scenes, and actions while adhering to linguistic habits. Given that most datasets are English-centric and Chinese possesses greater syntactic flexibility, this research addresses the specific hurdles of Chinese image description. Methods: Using the Flickr8k-cn and Flickr30k-cn datasets, we propose a model that dynamically decides whether to prioritize visual or textual information at each time step. The system utilizes a Convolutional Neural Network (CNN) encoder and a Long Short-Term Memory (LSTM) decoder integrated with a dual-attention mechanism. Results: Experiments demonstrate that this method outperforms existing Chinese abstract generation models. Conclusion: The proposed dual-attention approach significantly improves performance over benchmark models. Future work will incorporate more visual priors, such as object relationships and action categories, to further refine the quality of generated sentences.

Introduction

Image captioning—the task of generating a natural language description for a visual input—sits at the intersection of Computer Vision (CV) and Natural Language Processing (NLP). While English-based models have seen massive success, Chinese image captioning presents unique hurdles:

  1. Linguistic Complexity: Chinese has a more flexible syntax and complex lexical structure compared to English.
  2. The "Non-Visual Word" Trap: Standard attention mechanisms try to map every word to a specific region in an image. However, many functional words (like "is," "at," or "watching") are "non-visual." Forcing a visual anchor for these words introduces noise into the training process.
  3. Dataset Scarcity: Most open-source datasets are English-centric, leaving a gap in high-quality research for native Chinese generation.

Methodology

The proposed framework, DA-NIC, utilizes an encoder-decoder architecture enhanced by a novel Dual Attention Mechanism.

Encoder and Decoder

The Encoder uses a pre-trained ResNet-152 CNN to extract high-level visual features as 2048-dimensional vectors. The Decoder is an LSTM network that generates descriptions word-by-word. To handle Chinese text, the authors utilized the Jieba segmentation tool to create a word frequency dictionary of 8,560 unique tokens.

Overall architecture of the proposed Chinese image captioning system featuring a dual-attention decoder.

The Dual Attention Mechanism

Unlike standard spatial attention that only determines where to look in an image, the Dual Attention Mechanism adds a neural network classifier that determines if the model should look at the image at all.

As shown in the detailed module architecture below, the model uses a gating parameter ($\beta_t$) to balance visual features and textual context. If the word to be generated is highly visual (e.g., "dog," "ball"), the model leans on the image features. If the word is functional or semantic (e.g., "is," "beside"), the model relies more on the linguistic patterns learned from the text features.

Detailed view of the Dual Attention module within the LSTM decoder.

By allowing the decoder to "blind" itself to the image when generating functional grammar, the visual attention remains "sharp" for identifying objects and actions, avoiding the misleading gradients that plague simpler models.

Key Results

The DA-NIC model was evaluated on the Flickr8k-cn and Flickr30k-cn datasets. It consistently outperformed the standard Google NIC (Neural Image Caption) and the attention-based CNIC models across multiple metrics:

Metric (Flickr8k-cn) NIC (Baseline) CNIC (Attention) DA-NIC (Ours)
BLEU-4 (Accuracy) 32.7 33.5 37.1
CIDEr (Richness) 108.1 113.5 115.6
Rouge (Correlation) 61.1 62.5 66.4

Qualitative results showed that while a baseline model might describe a soccer game vaguely as "a group of people on a playground," the DA-NIC model correctly identified "little boys in red and white playing football," demonstrating a significantly higher level of descriptive detail.

Conclusion and Future Work

The dual attention mechanism effectively improves sentence quality by filtering out non-visual noise and better integrating textual creativity with visual evidence. However, the current model is limited to generating single descriptive sentences. Future iterations will aim to incorporate "visual prior information," such as explicit action categories and spatial relationships between objects, to achieve a more sophisticated "seeing the picture writing" effect.

Summary updated

Abstract

Objective: The Chinese description of images combines the two directions of computer vision and natural language processing. It is a typical representative of multi-mode and cross-domain problems with artificial intelligence algorithms. The image Chinese description model needs to output a Chinese description for each given test picture, describe the sentence requirements to conform to the natural language habits, and point out the important information in the image, covering the main characters, scenes, actions and other content. Since the current open source datasets are mostly in English, the research on the direction of image description is mainly in English. Chinese descriptions usually have greater flexibility in syntax and lexicalization, and the challenges of algorithm implementation are also large. Therefore, only a few people have studied image descriptions, especially Chinese descriptions. Methods: This study attempts to derive a model of image description generation from the Flickr8k-cn and Flickr30k-cn datasets. At each time period of the description, the model can decide whether to rely more on images or text information. The model captures more important information from the image to improve the richness and accuracy of the Chinese description of the image. The image description data set of this study is mainly composed of Chinese description sentences. The method consists of an encoder and a decoder. The encoder is based on a convolutional neural network. The decoder is based on a long-short memory network and is composed of a multi-modal summary generation network. Results: Experiments on Flickr8k-cn and Flickr30k-cn Chinese datasets show that the proposed method is superior to the existing Chinese abstract generation model. Conclusion: The method proposed in this paper is effective, and the performance has been greatly improved on the basis of the benchmark model. Compared with the existing Chinese abstract generation model, its performance is also superior. In the next step, more visual prior information will be incorporated into the model, such as the action category, the relationship between the object and the object, etc., to further improve the quality of the description sentence, and achieve the effect of “seeing the picture writing”.
BibTeX
@article{zhang2020application,
  author = {Yong Zhang and Jing Zhang},
  title = {{Application of Dual Attention Mechanism in Chinese Image Captioning}},
  journal = {Journal of Intelligent Learning Systems and Applications},
  year = {2020},
  doi = {10.4236/jilsa.2020.121002},
}
Figure