Standard vision-to-language models excel at objective inventory. Show them an image, and they will reliably output a literal description: "a dog sitting on the grass." While technically accurate, this style of language is socially sterile. When humans share or discuss images, we rarely just list visible objects. Instead, we use figurative language, inject personality, weave narratives, and express emotions—we might say, "Look at this majestic pup enjoying the sunshine!"
The research direction of subjective and stylized image captioning aims to bridge this factual-stylistic gap. The goal is to build models that can transition from robotic reporting to engaging, human-like prose without sacrificing the underlying visual truth.
The Core Dilemma: Grounding vs. Improvisation
The fundamental challenge in stylized captioning is balancing visual grounding with creative improvisation, a tension highlighted by Brett A. Halperin et al. in their study of narrative intelligence. If a model prioritizes factual accuracy too heavily, its outputs become formulaic and dry. Conversely, if it focuses entirely on stylistic flair, it risks hallucinating elements that are not present in the scene.
Compounding this challenge is a severe data scarcity problem. While massive datasets of literal image-caption pairs exist (such as MS-COCO), it is incredibly difficult to collect large-scale, high-quality datasets where images are paired with highly stylized or emotional text (e.g., romantic, humorous, or sarcastic descriptions). To bypass this, researchers have found creative ways to harvest natural data. For instance, Jianfu Chen et al. developed a method to automatically harvest millions of "Déjà Image-Captions"—verbatim descriptions that appear across multiple different web images—capturing rich, figurative expressions like "the sun is going to bed" that humans naturally use to describe a mood rather than an object list.
[ Traditional Captioning ] ---> "A dog on the grass." (Literal, Objective)
VS.
[ Stylized Captioning ] ---> "Look at this majestic pup enjoying the sunshine!" (Subjective, Engaging)
To scale these systems across diverse personas, however, models must learn to separate what is in an image from how it is expressed, using unpaired data.
Disentangling Content and Style
How can an AI learn to write a romantic or humorous caption for an image if it has never seen an example of a romantic caption paired with that specific image? The solution lies in architectural and latent disentanglement.
Factored Architectures
Early breakthroughs, such as StyleNet by Chuang Gan et al., introduced the concept of a "factored LSTM." Instead of using a single weight matrix to generate text, the model splits its parameters into a factual component (trained on standard image-caption pairs) and a stylistic component (trained on text-only corpora, like romantic novels or comedy scripts). When generating a caption, both matrices influence the hidden state, allowing the model to inject stylistic vocabulary while remaining anchored to the visual features.
Back-Translation and Contrastive Alignment
More recent frameworks eliminate the need for specialized architectures by using cycle-consistency and contrastive learning. As demonstrated in MSCap by Longteng Guo et al., a model can generate a stylized caption and then use a "back-translation" network to strip away the style, turning it back into a factual description. If the factual reconstruction matches the original image content, the stylized caption is deemed accurate.
Similarly, the ADS-Cap framework uses contrastive learning to align visual concepts from an image with textual concepts from an unpaired stylistic corpus in a shared latent space, ensuring that the stylistic choices remain semantically relevant to the scene.
+-------------+ +--------------------+ +------------------+
| Input Image | --> | Stylized Generator | --> | Humorous Caption |
+-------------+ +--------------------+ +------------------+
| |
v v
[Visual Grounding] [Back-Translation]
| |
v v
+--------------------+ +------------------+
| Contrastive Space | <-- | Factual Caption |
+--------------------+ +------------------+
Reward Functions for the "Vibe"
Standard language models are trained using cross-entropy loss, which penalizes any deviation from a ground-truth reference text. This training objective actively discourages creativity. To generate truly engaging text, researchers use adversarial training and reinforcement learning to reward models for capturing a specific "vibe."
In Show, Reward and Tell, Jing Wang et al. proposed a dual-critic architecture within an adversarial framework:
- The Visual Relevance Critic: Ensures the generated text is grounded in the image sequence.
- The Style Critic: Evaluates whether the text matches the target narrative distribution.
This idea of optimizing for engagement rather than token overlap is pushed further when models interact with social environments. For example, Yuchong Sun et al. developed ViCo, a framework that uses a "likes"-based reward model trained on human social media interactions. By optimizing via Reinforcement Learning from Human Feedback (RLHF), the model learns to output subjective, witty, or emotional comments that resonate with users, moving far beyond dry video summaries. A similar shift toward high-engagement language is seen in work by Mark Dras et al. (ATTEND-GAN) and Yuyan Chen et al. (HOTVCOM), which focus on generating "buzzworthy" social media commentary.
Inferring the Unseen: Narrative Intelligence
When humans look at a sequence of photos—such as a family vacation album—we don't just describe each frame in isolation. We construct a cohesive story arc, inferring character motivations, backstories, and off-screen context.
[ Image 1: Packing bags ] ---> [ Image 2: Airport terminal ] ---> [ Image 3: Beach sunset ]
|
(Inferential Leap)
v
"After months of anticipation, our journey finally began as we boarded the flight to paradise."
As Licheng Yu et al. and Gunhee Kim et al. observed, effective storytelling requires a shift from frame-by-frame captioning to event-level significance. To achieve this, models must make inferential leaps beyond the visible pixels.
Graph-Based Reasoning and Commonsense
To help models make these creative leaps, researchers inject external knowledge. The Imagine-Reason-Write (IRW) framework by Chunpu Xu et al. explicitly predicts "imaginary concepts" that are likely to happen between frames. It uses a triple-graph reasoning approach—combining scene graphs, event graphs, and commonsense knowledge—to link visual cues to abstract narrative concepts. Similarly, Pengcheng Yang et al. use a commonsense-driven generative model to fill in the blanks of a story, allowing the AI to discuss character feelings and potential consequences that aren't explicitly visible.
Explicit Emotion and Context Conditioning
Narratives also require structural flow. Models like ViNTER, developed by Kohei Uehara et al., accept an explicit "emotion arc" (e.g., neutral $\rightarrow$ sadness $\rightarrow$ joy) as an input constraint alongside the images. This allows the model to control the subjective tone and pacing of the prose across a sequence.
When applied to specialized domains, this contextual awareness allows models to adapt to highly specific human communication games. This ranges from weaving named entities into Wikipedia articles (Khanh Nguyen et al.), to generating sensational sports commentary (Hassan Mkhallati et al.), to formulating open-ended, conversational questions that spark social media engagement (Min-Hsuan Yeh et al.).
Ultimately, subjective and stylized image captioning transforms vision-language models from passive observers into active, personality-driven storytellers that communicate more like humans and less like machines.
Key Foundational Work
Déjà Image-Captions: A Corpus of Expressive Descriptions in Repetition (2015) — contributed 1 concept elements to this direction.
Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015) — contributed 1 concept elements to this direction.
Visual Storytelling (2016) — contributed 1 concept elements to this direction.