Abstract
Automatic image captioning is a critical task in computer vision with applications ranging from accessibility for the visually impaired to automated content indexing. While deep learning models have shown significant promise, they often suffer from overfitting due to limited training data, leading them to rely on spurious correlations or "shortcuts" within images. This paper proposes a novel data augmentation method that leverages attention mechanisms to identify and target these overfitted regions. By applying content-aware cropping and masking based on attention maps, the proposed method forces the model to learn more robust features. Experimental results on the MSCOCO dataset demonstrate that this approach significantly improves captioning performance across standard evaluation metrics.
The Problem of "Shortcut" Learning
Deep learning models for image captioning frequently suffer from overfitting, particularly when training datasets are limited. A specific type of overfitting occurs when a model learns "shortcuts." For example, if many "car" images in a dataset feature visible license plates, the model might learn to associate the small rectangular shape of a license plate with the word "car," ignoring the rest of the vehicle.
Existing solutions like random cropping or "Cutout" (masking random squares) are "blind" to the image content. They might accidentally remove the primary subject entirely or miss the overfitted region altogether. The authors sought a way to make data augmentation "content-aware."
Proposed Method: Attention-Guided Augmentation
The authors introduce a pipeline that makes data augmentation "content-aware" by using the model's own focus as a guide.
The process begins with generating attention maps ($I_{att}$) using a pre-trained "Show, Attend and Tell" model. These maps indicate which pixels are most influential for generating specific words. To isolate these regions, K-means clustering is applied to the attention map, segmenting the image into high-focus and low-focus areas.
Targeted Augmentation Techniques
Based on the identified high-attention regions, two primary augmentation strategies are employed:
- Attention-based Cropping: The system identifies the bounding box of the high-attention region and creates new training samples by cropping the image to focus on these areas or ensuring they are preserved while varying the background.
- Attention-based Masking: Instead of masking a random square, the system places a mask directly over the high-attention region.
Crucially, the size of the mask scales dynamically with the size of the attention area, ensuring the augmentation is proportional to the object's scale. By masking the very thing the model is looking at most intensely, the model is forced to look at the surrounding context (e.g., the wheels and body of a car instead of just the license plate) to generate the correct caption.
Experimental Results
The method was evaluated on the MSCOCO '14 dataset, a standard benchmark in the field. The proposed attention-based augmentation outperformed baseline models and standard random augmentation techniques across all standard metrics:
- CIDEr: 103.73 (vs. 99.64 baseline)
- BLEU-4: 31.8
- METEOR: 26.09
Qualitative analysis revealed that the model became better at identifying objects based on their overall structure. In one example, a baseline model misidentified a motorcycle as a "skateboard" because it over-indexed on a small part of the frame. The proposed model, having been trained with attention-based masks, correctly identified the "motorcycle" by looking at the broader structure of the vehicle.
Conclusion
By using attention maps as a diagnostic tool to identify overfitted regions, this research provides a more surgical approach to data augmentation. The results indicate that "breaking" false correlations through targeted masking and cropping significantly enhances the generalization capabilities of image captioning models. Future work will explore more sophisticated clustering algorithms and the optimal balance of augmentation intensity.
Summary updated