Similar Scenes Arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning

Guodun Li, Yuchen Zhai, Zehao Lin, Yin Zhang

TL;DR

Stylized image captioning—generating descriptions that are "humorous" or "romantic" rather than just factual—is hampered by a massive shortage of paired training data. Researchers have developed a novel framework called Extract-Retrieve-Generate (ERG) that "grafts" emotional style phrases from small datasets onto large-scale factual datasets, creating high-quality synthetic training pairs that significantly boost model performance.

What Problem Does This Paper Address?

Standard image captioning models are great at saying "a dog sits on the grass," but they lack "personality." Stylized image captioning aims to add flavor (e.g., "a loyal companion rests peacefully on the emerald lawn").

The bottleneck is data. While we have millions of factual captions (like MSCOCO), we only have a few thousand stylized ones. Writing these manually is expensive and slow. Most researchers try "unsupervised" methods to bridge this gap, but these often struggle to balance factual accuracy with stylistic flair. This paper tackles the problem from a different angle: Data Augmentation. How can we automatically create a massive, high-quality dataset of stylized captions without hiring thousands of poets and comedians?

Methods

The authors propose the Extract-Retrieve-Generate (ERG) framework, inspired by the psychological "context effect"—the idea that similar scenes evoke similar emotions.

  1. Emotional Signal Extractor: Using a BERT-based classifier, the system identifies which specific words in a stylized sentence carry the "emotion." By looking at attention weights, it isolates style phrases (e.g., "enjoying the beauty of nature") from the factual content.
  2. Multi-Modal Scene Retriever: The system takes a factual image (one that lacks a stylized caption) and looks for "similar" scenes in the small stylized dataset. It uses a pre-trained model called OSCAR to perform four types of searches: image-to-image, image-to-text, text-to-image, and text-to-text.
  3. Emotion-aware Caption Generator: Finally, a GPT-2 model acts as a "grafter." It takes the factual description of the new image and the extracted style phrase from the retrieved similar scene to generate a brand-new, fluent, stylized caption.

Crucially, the authors use three "quality control" filters: the generated caption must pass a style classifier, maintain low perplexity (be grammatically fluent), and come from a highly similar retrieved scene.

Datasets

The study utilizes several standard benchmarks:

  • FlickrStyle10K: Contains 10,000 images with humorous and romantic captions (though only 7,000 are currently accessible).
  • SentiCap: Focuses on positive and negative sentiment captions.
  • MSCOCO & Flickr30K: Used as the "Large-Scale Factual Corpus"—the source of images that need stylized "upgrades."

Key Insights and Ideas

  • Style is Localized: One of the most important "aha moments" is that style often resides in specific phrases rather than the entire sentence structure. By isolating these "emotional signals," the model can transplant them more effectively.
  • The Power of Cross-Modal Retrieval: The authors found that retrieving similar scenes isn't just about looking at pixels (image-to-image). Often, comparing the factual text of one image to the stylized text of another (text-to-text) yields much more relevant stylistic matches.
  • Model Agnosticism: This isn't just a new model; it's a new way to train any model. The augmented data improved everything from "old-school" LSTMs (NIC) to cutting-edge Transformers (VinVL).

Key Results

The ERG framework provided a massive boost across the board:

  • Style Accuracy: For the "Humor" style, a standard NIC model jumped from 82.8% to 97.5% accuracy in style classification after being trained on augmented data.
  • Factual Relevance: The CIDEr score (a measure of how well the caption matches the image content) improved significantly. For "Positive" sentiment, the NIC model's CIDEr score rose by 39.2%.
  • State-of-the-Art (SOTA): When applied to a Self-Attention Network (SAN), the framework outperformed previous specialized models like MemCap and MSCap by a substantial margin in both stylishness and relevance.

Limitations and Future Work

While the results are impressive, the authors note a few hurdles:

  • Noise in Retrieval: Even with multi-modal retrieval, the "similar" scene found might occasionally be a mismatch, leading to slightly awkward "grafts."
  • The "Upper Bound" of Data: The researchers found that while more data helps, there is a point of diminishing returns. After a certain volume, adding more synthetic data (which inherently contains some noise) can actually cause a slight dip in factual relevance.
  • Future Directions: The authors suggest that this "Extract-Retrieve-Generate" pipeline could be adapted for even more diverse styles beyond just humor, romance, and sentiment.

Summary updated

Abstract

Stylized image captioning systems aim to generate a caption not only semantically related to a given image but also consistent with a given style description. One of the biggest challenges with this task is the lack of sufficient paired stylized data. Many studies focus on unsupervised approaches, without considering from the perspective of data augmentation. We begin with the observation that people may recall similar emotions when they are in similar scenes, and often express similar emotions with similar style phrases, which underpins our data augmentation idea. In this paper, we propose a novel Extract-Retrieve-Generate data augmentation framework to extract style phrases from small-scale stylized sentences and graft them to large-scale factual captions. First, we design the emotional signal extractor to extract style phrases from small-scale stylized sentences. Second, we construct the plugable multi-modal scene retriever to retrieve scenes represented with pairs of an image and its stylized caption, which are similar to the query image or caption in the large-scale factual data. In the end, based on the style phrases of similar scenes and the factual description of the current scene, we build the emotion-aware caption generator to generate fluent and diversified stylized captions for the current scene. Extensive experimental results show that our framework can alleviate the data scarcity problem effectively. It also significantly boosts the performance of several existing image captioning models in both supervised and unsupervised settings, which outperforms the state-of-the-art stylized image captioning methods in terms of both sentence relevance and stylishness by a substantial margin.
BibTeX
@inproceedings{li2021similar,
  author = {Guodun Li and Yuchen Zhai and Zehao Lin and Yin Zhang},
  title = {{Similar Scenes Arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=6Is6ZatMArn},
}

Loading related content...

Figure