Introduction
Stylized image captioning aims to generate descriptions that are not only semantically accurate but also possess a specific personality, such as being "humorous" or "romantic." While standard models can effectively describe a scene factually (e.g., "a dog sits on the grass"), they often lack the "flavor" found in human-written descriptions (e.g., "a loyal companion rests peacefully on the emerald lawn").
The primary bottleneck in this field is the scarcity of paired training data. While millions of factual captions exist (e.g., MSCOCO), stylized datasets like FlickrStyle10K are much smaller. Manually creating these datasets is prohibitively expensive. This paper introduces the Extract-Retrieve-Generate (ERG) framework, a data augmentation approach inspired by the psychological "context effect"—the observation that similar scenes often evoke similar emotions and expressions.
The ERG Framework
The ERG framework "grafts" emotional style phrases from small stylized datasets onto large-scale factual datasets to create high-quality synthetic training pairs. It is built on the insight that style often resides in specific phrases rather than the entire sentence structure.
The pipeline consists of three main components:
- Emotional Signal Extractor: This component identifies style-carrying words within a stylized sentence. Using a BERT-based classifier and analyzing attention weights, the system isolates style phrases (e.g., "enjoying the beauty of nature") from the factual content.
- Multi-Modal Scene Retriever: For a given factual image, the system searches for similar scenes in the small stylized dataset. It employs a pre-trained OSCAR model to perform cross-modal searches (image-to-image, image-to-text, text-to-image, and text-to-text), ensuring a robust match between the query scene and potential style sources.
- Emotion-aware Caption Generator: A GPT-2 model serves as the "grafter," combining the factual description of the target image with the extracted style phrase from the retrieved similar scene to generate a fluent, stylized caption.
To ensure quality, the framework applies three filters: the generated caption must pass a style classifier, maintain grammatical fluency (low perplexity), and originate from a highly similar retrieved scene.
Experimental Results
The researchers evaluated ERG on standard benchmarks including FlickrStyle10K and SentiCap, using MSCOCO and Flickr30K as the large-scale factual corpora. The results demonstrate that ERG significantly enhances both stylishness and relevance across various base models (e.g., NIC, SAN, VinVL), proving the framework is model-agnostic.
- Style Accuracy: Training a standard NIC model on ERG-augmented data improved its "Humor" style classification accuracy from 82.8% to 97.5%.
- Factual Relevance: The CIDEr score, which measures alignment with image content, rose by 39.2% for the "Positive" sentiment style in the NIC model.
- State-of-the-Art (SOTA) Performance: When integrated with a Self-Attention Network (SAN), the framework outperformed existing state-of-the-art models like MemCap and MSCap by a substantial margin in both sentence relevance and stylishness.
Conclusion
The ERG framework effectively addresses the data scarcity problem in stylized image captioning through intelligent data augmentation. By recognizing that style is often localized in specific phrases and leveraging cross-modal retrieval of similar scenes, the system can generate diverse and high-quality training data. While retrieval noise and data upper bounds remain challenges, this "Extract-Retrieve-Generate" pipeline offers a promising path for scaling stylized AI beyond current sentiment and personality categories.
Summary updated