Image Captioning with Data Augmentation Using Cropping and Mask Based on Attention Image

Kiyohiko Iwamura, Jun Younes Louhi Kasahara, Alessandro Moro, Atsushi Yamashita, Hajime AsamaJournal of the Japan Society for Precision Engineering 2020 · DOI

Abstract

Automatic image captioning is a critical task in computer vision with applications ranging from accessibility for the visually impaired to automated content indexing. While deep learning models have shown significant promise, they often suffer from overfitting due to limited training data, leading them to rely on spurious correlations or "shortcuts" within images. This paper proposes a novel data augmentation method that leverages attention mechanisms to identify and target these overfitted regions. By applying content-aware cropping and masking based on attention maps, the proposed method forces the model to learn more robust features. Experimental results on the MSCOCO dataset demonstrate that this approach significantly improves captioning performance across standard evaluation metrics.

The Problem of "Shortcut" Learning

Deep learning models for image captioning frequently suffer from overfitting, particularly when training datasets are limited. A specific type of overfitting occurs when a model learns "shortcuts." For example, if many "car" images in a dataset feature visible license plates, the model might learn to associate the small rectangular shape of a license plate with the word "car," ignoring the rest of the vehicle.

Existing solutions like random cropping or "Cutout" (masking random squares) are "blind" to the image content. They might accidentally remove the primary subject entirely or miss the overfitted region altogether. The authors sought a way to make data augmentation "content-aware."

Proposed Method: Attention-Guided Augmentation

The authors introduce a pipeline that makes data augmentation "content-aware" by using the model's own focus as a guide.

The proposed multi-step pipeline for attention-based data augmentation.

The process begins with generating attention maps ($I_{att}$) using a pre-trained "Show, Attend and Tell" model. These maps indicate which pixels are most influential for generating specific words. To isolate these regions, K-means clustering is applied to the attention map, segmenting the image into high-focus and low-focus areas.

Targeted Augmentation Techniques

Based on the identified high-attention regions, two primary augmentation strategies are employed:

  1. Attention-based Cropping: The system identifies the bounding box of the high-attention region and creates new training samples by cropping the image to focus on these areas or ensuring they are preserved while varying the background.
  2. Attention-based Masking: Instead of masking a random square, the system places a mask directly over the high-attention region.
Example of attention-based masking where a specific region of interest is obscured to force the model to consider broader context.

Crucially, the size of the mask scales dynamically with the size of the attention area, ensuring the augmentation is proportional to the object's scale. By masking the very thing the model is looking at most intensely, the model is forced to look at the surrounding context (e.g., the wheels and body of a car instead of just the license plate) to generate the correct caption.

Experimental Results

The method was evaluated on the MSCOCO '14 dataset, a standard benchmark in the field. The proposed attention-based augmentation outperformed baseline models and standard random augmentation techniques across all standard metrics:

  • CIDEr: 103.73 (vs. 99.64 baseline)
  • BLEU-4: 31.8
  • METEOR: 26.09

Qualitative analysis revealed that the model became better at identifying objects based on their overall structure. In one example, a baseline model misidentified a motorcycle as a "skateboard" because it over-indexed on a small part of the frame. The proposed model, having been trained with attention-based masks, correctly identified the "motorcycle" by looking at the broader structure of the vehicle.

Conclusion

By using attention maps as a diagnostic tool to identify overfitted regions, this research provides a more surgical approach to data augmentation. The results indicate that "breaking" false correlations through targeted masking and cropping significantly enhances the generalization capabilities of image captioning models. Future work will explore more sophisticated clustering algorithms and the optimal balance of augmentation intensity.

Summary updated

Abstract

Automatic image captioning has various important applications such as the depiction of contents for the visually impaired. Most approaches use Deep Learning and have achieved remarkable results. However there are still some unresolved issues. One of them is the overfitting of the trained model to specific images, usually caused by limited training dataset sizes. In order to augment the training dataset size in such scenarios, previous researches proposed data augmentation using random cropping or mask. However, those do not specifically target overfitted regions in images and, therefore, may remove areas in images that are needed to generate captions and lower performance. In this study, we propose a novel data augmentation method that targets specifically regions in images subject to overfitting by using attention. Experimental results show that the proposed method allows generation of better image captions.
BibTeX
@article{iwamura2020image,
  author = {Kiyohiko Iwamura and Jun Younes Louhi Kasahara and Alessandro Moro and Atsushi Yamashita and Hajime Asama},
  title = {{Image Captioning with Data Augmentation Using Cropping and Mask Based on Attention Image}},
  journal = {Journal of the Japan Society for Precision Engineering},
  year = {2020},
  doi = {10.2493/jjspe.86.904},
}
Figure