Abstract
Objective: The Chinese description of images combines computer vision and natural language processing, representing a multi-mode and cross-domain challenge for artificial intelligence. This study aims to develop an image captioning model that generates natural Chinese descriptions, identifying key characters, scenes, and actions while adhering to linguistic habits. Given that most datasets are English-centric and Chinese possesses greater syntactic flexibility, this research addresses the specific hurdles of Chinese image description. Methods: Using the Flickr8k-cn and Flickr30k-cn datasets, we propose a model that dynamically decides whether to prioritize visual or textual information at each time step. The system utilizes a Convolutional Neural Network (CNN) encoder and a Long Short-Term Memory (LSTM) decoder integrated with a dual-attention mechanism. Results: Experiments demonstrate that this method outperforms existing Chinese abstract generation models. Conclusion: The proposed dual-attention approach significantly improves performance over benchmark models. Future work will incorporate more visual priors, such as object relationships and action categories, to further refine the quality of generated sentences.
Introduction
Image captioning—the task of generating a natural language description for a visual input—sits at the intersection of Computer Vision (CV) and Natural Language Processing (NLP). While English-based models have seen massive success, Chinese image captioning presents unique hurdles:
- Linguistic Complexity: Chinese has a more flexible syntax and complex lexical structure compared to English.
- The "Non-Visual Word" Trap: Standard attention mechanisms try to map every word to a specific region in an image. However, many functional words (like "is," "at," or "watching") are "non-visual." Forcing a visual anchor for these words introduces noise into the training process.
- Dataset Scarcity: Most open-source datasets are English-centric, leaving a gap in high-quality research for native Chinese generation.
Methodology
The proposed framework, DA-NIC, utilizes an encoder-decoder architecture enhanced by a novel Dual Attention Mechanism.
Encoder and Decoder
The Encoder uses a pre-trained ResNet-152 CNN to extract high-level visual features as 2048-dimensional vectors. The Decoder is an LSTM network that generates descriptions word-by-word. To handle Chinese text, the authors utilized the Jieba segmentation tool to create a word frequency dictionary of 8,560 unique tokens.
The Dual Attention Mechanism
Unlike standard spatial attention that only determines where to look in an image, the Dual Attention Mechanism adds a neural network classifier that determines if the model should look at the image at all.
As shown in the detailed module architecture below, the model uses a gating parameter ($\beta_t$) to balance visual features and textual context. If the word to be generated is highly visual (e.g., "dog," "ball"), the model leans on the image features. If the word is functional or semantic (e.g., "is," "beside"), the model relies more on the linguistic patterns learned from the text features.
By allowing the decoder to "blind" itself to the image when generating functional grammar, the visual attention remains "sharp" for identifying objects and actions, avoiding the misleading gradients that plague simpler models.
Key Results
The DA-NIC model was evaluated on the Flickr8k-cn and Flickr30k-cn datasets. It consistently outperformed the standard Google NIC (Neural Image Caption) and the attention-based CNIC models across multiple metrics:
| Metric (Flickr8k-cn) | NIC (Baseline) | CNIC (Attention) | DA-NIC (Ours) |
|---|---|---|---|
| BLEU-4 (Accuracy) | 32.7 | 33.5 | 37.1 |
| CIDEr (Richness) | 108.1 | 113.5 | 115.6 |
| Rouge (Correlation) | 61.1 | 62.5 | 66.4 |
Qualitative results showed that while a baseline model might describe a soccer game vaguely as "a group of people on a playground," the DA-NIC model correctly identified "little boys in red and white playing football," demonstrating a significantly higher level of descriptive detail.
Conclusion and Future Work
The dual attention mechanism effectively improves sentence quality by filtering out non-visual noise and better integrating textual creativity with visual evidence. However, the current model is limited to generating single descriptive sentences. Future iterations will aim to incorporate "visual prior information," such as explicit action categories and spatial relationships between objects, to achieve a more sophisticated "seeing the picture writing" effect.
Summary updated