Contextualized Image Captioning

35 concepts25 papers2 proposals

Standard image captioning models operate under a simple assumption: the meaning of an image is entirely contained within its pixels. When presented with a photograph, a traditional model extracts visual features and generates a literal description, such as "a man in a suit speaking at a podium."

While factually accurate, this "one-size-fits-all" approach fails in real-world applications. If that same image appears in a news article about a policy debate, a Wikipedia entry about a specific politician, or an economics textbook, its communicative purpose changes entirely. The literal description lacks the necessary real-world entities, socio-political significance, and thematic relevance required by the surrounding text.

To bridge this gap, Contextualized Image Captioning shifts the objective from generic object enumeration to narrative-grounded description. By treating surrounding text, metadata, or user intent as a primary input alongside the image, these systems generate descriptions that are dynamically tailored to their specific environment.


The Information Bottleneck of Pure Vision

Why do standard vision models struggle when deployed "in the wild"? As Tran et al. observe, generic captioning systems routinely neglect specific real-world entities like celebrities and landmarks, which severely hinders common-sense understanding.

From a purely visual standpoint, many entities are indistinguishable. A photo of a political summit looks remarkably similar to a photo of a corporate board meeting. As Batra et al. point out, visual features alone cannot resolve the specific identities, locations, or broader socio-political significance of a scene.

Furthermore, professional journalism and editorial media rarely use captions to describe what is obviously visible. Instead, as Hollink et al. demonstrate through the ION corpus, news captions provide editorial framing, quotes, and thematic summaries. The text and the image exist in a non-descriptive, complementary relationship where the text provides the interpretive key to the image.


Resolving Ambiguity via Textual Grounding

To generate context-rich captions, models must treat the accompanying text as a primary source of knowledge. Rather than hallucinating or guessing names and dates from pixels, models extract this high-density information directly from the document narrative.

The Copy Mechanism and Template Filling

One of the earliest and most effective intuitions in this field is that the words needed for a contextual caption already exist in the surrounding text. Rather than generating a caption entirely from a fixed vocabulary, models use a pointer-generator or copy mechanism to dynamically pull out-of-vocabulary named entities directly from the article.

For example, Biten introduced a two-stage framework that leverages associated news articles to insert specific people or locations into generated templates. Similarly, the Visual News Captioner developed by Liu et al. fuses visual data with article text, using a pointer mechanism to copy precise names (such as specific athletes or politicians) directly into the output.

Context-Aware Pretraining

To make this integration seamless, models can be pretrained to explicitly align visual regions with textual entities. In the context of Wikipedia articles, Nguyen et al. proposed Masked Named Entity Modeling (MNEM). This strategy forces the model to predict hidden named entities by cross-referencing the remaining text with the image, training the network to understand exactly how a visual element corresponds to a specific textual mention. As Kalarani et al. demonstrate, explicitly incorporating this contextual information during pretraining significantly prevents the model from isolating visual features, leading to much higher domain-specific accuracy.

[ Surrounding Article Text ] ───► [ Context Fusion / Attention ] ◄─── [ Image Pixels ]
                                               │
                                               ▼
                                  [ Pointer-Generator / Copy ]
                                               │
                                               ▼
                                  "Bernie Sanders speaks at..."

One Image, Multiple Narratives

A critical insight of contextualized captioning is that an image does not have a single correct caption. If an image is embedded in a complex document, it might relate to multiple distinct themes discussed across different paragraphs.

To handle this, Mao et al. introduced Controllable Contextualized Image Captioning (Ctrl-CIC). Their framework allows users to highlight specific segments of a surrounding text context (such as a particular paragraph in a Wikipedia article). Lightweight controllers then recalibrate the model's internal embeddings, directing the visual narrative to prioritize the highlighted aspect of the background information.

This flexibility is equally vital when generating alternative text (alt-text) for digital publishing. As Shen et al. show with the AltGen pipeline, integrating adjacent paragraphs ensures that the exact same image receives a highly technical description when placed in a textbook, but a more literary, narrative description when placed in a novel.

In scientific publishing, this mechanism is pushed even further. The SciXGen dataset introduced by Hong et al. demonstrates that captions for complex tables and figures cannot be generated in isolation; models must read the surrounding body text to resolve domain-specific abbreviations and interpret data trends that are only explained in the narrative.


Pragmatics and Goal-Oriented Adaptation

When humans describe images, they naturally adapt their language based on the listener's needs. Contextualized captioning formalizes this behavior using pragmatic frameworks and goal-oriented optimization.

Issue-Sensitive Captioning via RSA

Instead of retraining massive neural networks for every new context, Nie et al. applied the Rational Speech Acts (RSA) model to image captioning. This framework treats captioning as a game of pragmatic communication: the model evaluates candidate descriptions and selects the one that best addresses a specific "issue" or question relevant to the user's current situation, ensuring situational relevance without architectural overhead.

Captioning with a Purpose

For specialized user groups, such as blind and low-vision (BLV) individuals, a description's quality is inextricably linked to its utility. As Fisch et al. argue in their CapWAP (Captioning with A Purpose) framework, generic human-annotated captions often omit high-utility details (like a bus route number or a specific product ingredient). By optimizing captions using reinforcement learning with a question-answering model in the loop, systems learn to preserve the exact details needed to answer downstream user queries.

This task-centric focus is echoed by Dognin et al. in the VizWiz challenge, where combining rotation-invariant optical character recognition (OCR) with copy mechanisms allowed models to prioritize fine-grained text on packaging over broad scene descriptions.

Ultimately, as Kreiss et al. and Zur et al. emphasize, evaluating these models requires moving away from "context-blind" metrics. A truly effective vision-language model does not just see the world—it understands where its vision fits into the broader human conversation.

Key Foundational Work

A Corpus of Images and Text in Online News (2016) — contributed 2 concept elements to this direction.
Control Structures for Incorporating Picture-Specific Context in Image Interpretation (1995) — contributed 1 concept elements to this direction.
Rich Image Captioning in the Wild (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_context_dependent_image_captioning_c09e53f4,
  title = {context-dependent image captioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/context-dependent-image-captioning/rdr_c09e53f41bcc458eb8070a9e3d3924f1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_context_dependent_image_captioning_c09e53f4_v1,
  title = {context-dependent image captioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/context-dependent-image-captioning/rdr_c09e53f41bcc458eb8070a9e3d3924f1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake