Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning

Guodun Li, Yuchen Zhai, Zehao Lin, Yin Zhang

Introduction

Stylized image captioning aims to generate descriptions that are not only semantically accurate but also possess a specific personality, such as being "humorous" or "romantic." While standard models can effectively describe a scene factually (e.g., "a dog sits on the grass"), they often lack the "flavor" found in human-written descriptions (e.g., "a loyal companion rests peacefully on the emerald lawn").

The primary bottleneck in this field is the scarcity of paired training data. While millions of factual captions exist (e.g., MSCOCO), stylized datasets like FlickrStyle10K are much smaller. Manually creating these datasets is prohibitively expensive. This paper introduces the Extract-Retrieve-Generate (ERG) framework, a data augmentation approach inspired by the psychological "context effect"—the observation that similar scenes often evoke similar emotions and expressions.

Conceptual motivation: similar scenes evoke similar emotional expressions.

The ERG Framework

The ERG framework "grafts" emotional style phrases from small stylized datasets onto large-scale factual datasets to create high-quality synthetic training pairs. It is built on the insight that style often resides in specific phrases rather than the entire sentence structure.

The Extract-Retrieve-Generate (ERG) framework architecture.

The pipeline consists of three main components:

  1. Emotional Signal Extractor: This component identifies style-carrying words within a stylized sentence. Using a BERT-based classifier and analyzing attention weights, the system isolates style phrases (e.g., "enjoying the beauty of nature") from the factual content.
  2. Multi-Modal Scene Retriever: For a given factual image, the system searches for similar scenes in the small stylized dataset. It employs a pre-trained OSCAR model to perform cross-modal searches (image-to-image, image-to-text, text-to-image, and text-to-text), ensuring a robust match between the query scene and potential style sources.
  3. Emotion-aware Caption Generator: A GPT-2 model serves as the "grafter," combining the factual description of the target image with the extracted style phrase from the retrieved similar scene to generate a fluent, stylized caption.

To ensure quality, the framework applies three filters: the generated caption must pass a style classifier, maintain grammatical fluency (low perplexity), and originate from a highly similar retrieved scene.

Experimental Results

The researchers evaluated ERG on standard benchmarks including FlickrStyle10K and SentiCap, using MSCOCO and Flickr30K as the large-scale factual corpora. The results demonstrate that ERG significantly enhances both stylishness and relevance across various base models (e.g., NIC, SAN, VinVL), proving the framework is model-agnostic.

  • Style Accuracy: Training a standard NIC model on ERG-augmented data improved its "Humor" style classification accuracy from 82.8% to 97.5%.
  • Factual Relevance: The CIDEr score, which measures alignment with image content, rose by 39.2% for the "Positive" sentiment style in the NIC model.
  • State-of-the-Art (SOTA) Performance: When integrated with a Self-Attention Network (SAN), the framework outperformed existing state-of-the-art models like MemCap and MSCap by a substantial margin in both sentence relevance and stylishness.

Conclusion

The ERG framework effectively addresses the data scarcity problem in stylized image captioning through intelligent data augmentation. By recognizing that style is often localized in specific phrases and leveraging cross-modal retrieval of similar scenes, the system can generate diverse and high-quality training data. While retrieval noise and data upper bounds remain challenges, this "Extract-Retrieve-Generate" pipeline offers a promising path for scaling stylized AI beyond current sentiment and personality categories.

Summary updated

Abstract

Stylized image captioning systems aim to generate a caption not only semantically related to a given image but also consistent with a given style description. One of the biggest challenges with this task is the lack of sufficient paired stylized data. Many studies focus on unsupervised approaches, without considering from the perspective of data augmentation. We begin with the observation that people may recall similar emotions when they are in similar scenes, and often express similar emotions with similar style phrases, which underpins our data augmentation idea. In this paper, we propose a novel Extract-Retrieve-Generate data augmentation framework to extract style phrases from small-scale stylized sentences and graft them to large-scale factual captions. First, we design the emotional signal extractor to extract style phrases from small-scale stylized sentences. Second, we construct the plugable multi-modal scene retriever to retrieve scenes represented with pairs of an image and its stylized caption, which are similar to the query image or caption in the large-scale factual data. In the end, based on the style phrases of similar scenes and the factual description of the current scene, we build the emotion-aware caption generator to generate fluent and diversified stylized captions for the current scene. Extensive experimental results show that our framework can alleviate the data scarcity problem effectively. It also significantly boosts the performance of several existing image captioning models in both supervised and unsupervised settings, which outperforms the state-of-the-art stylized image captioning methods in terms of both sentence relevance and stylishness by a substantial margin.
BibTeX
@inproceedings{li2021similar,
  author = {Guodun Li and Yuchen Zhai and Zehao Lin and Yin Zhang},
  title = {{Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=6Is6ZatMArn},
}
Figure