Global Narrative Coherence for Sequential Visual Storytelling

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2128 papers141 conceptsfrozen 2026-10-04

Traditional computer vision models excel at describing what is in a single image, but they often fail when tasked with describing a sequence. If you show a standard captioning model five photos of a wedding, it might produce five accurate but disjointed sentences: "a woman in a white dress," "people sitting in chairs," "a cake on a table." To a human, this is not a story; it is a list.

The research direction of Visual Storytelling seeks to bridge this gap. It moves beyond literal description toward narrative synthesis, ensuring that a sequence of images (or a sequence of sentences generated from images) possesses a logical "thread" that connects individual moments into a cohesive whole.

The Problem of Disjointed Narratives

The fundamental challenge identified across this field is that a "bag of captions" lacks causal and temporal logic. As noted in Expressing an Image Stream with a Sequence of Natural Sentences by Cesc Chunseong Park and Gunhee Kim, isolated descriptions fail to capture the narrative structure inherent in sequential data.

This disjointedness stems from a lack of global context. When a model generates a sentence for the third image in a stream, it must remember what happened in the first two to avoid repetition and maintain entity consistency. Without this, models produce "boring" or repetitive stories that optimize for word-overlap metrics like BLEU but fail to convey a meaningful arc.

Hierarchical Planning and Blueprints

To solve this, researchers have moved away from "flat" models that generate text word-by-word in favor of hierarchical architectures that plan before they write.

Qiuyuan Huang, Aslı Çelikyılmaz, and colleagues proposed a Hierarchically Structured Reinforcement Learning (HSRL) framework in Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation. Their model uses a "Manager" to set high-level topical goals for each image and a "Worker" to generate the specific sentences. This division of labor ensures that the narrative stays on track.

Similarly, Danyang Liu introduced a "blueprint" approach in Visual Storytelling with Question-Answer Plans. By forcing the model to first generate a sequence of question-answer pairs about the images, the system creates a logical scaffolding that grounds the final narrative in specific visual evidence.

Grounding Stories in Structure and Knowledge

Another recurring strategy is to transform raw pixels into structured representations like Scene Graphs. In Storytelling from an Image Stream Using Scene Graphs, Ruize Wang and Zhongyu Wei argue that treating images as "black box" vectors is insufficient. By explicitly modeling the relationships between objects (e.g., "man-holding-cake"), models can better track how these relationships evolve over time.

For more complex tasks like recipe generation, Khyathi Raghavi Chandu and Alan W. Black utilized Finite State Machines (FSMs) to model the "phases" of a process, ensuring that "chopping" logically precedes "frying" in Storyboarding of Recipes: Grounded Contextual Generation.

When visual evidence is sparse, models must rely on "common sense." Ting-Hao Kenneth Huang and Chao-Chun Hsu developed KG-Story, which enriches image features with external Knowledge Graphs. This allows the model to infer causal links—such as knowing that a photo of a trophy likely implies a previous "competition"—that aren't explicitly visible in the frame.

Story Visualization: The Inverse Challenge

The quest for coherence also applies to the inverse task: Story Visualization, or generating a sequence of images from a multi-sentence story.

Standard text-to-image models often suffer from "character drift," where a protagonist's appearance changes from one frame to the next. Yitong Li 0001, Zhe Gan, and Yu Cheng addressed this with StoryGAN, which uses a global context encoder to track the narrative state across the entire sequence.

To evaluate these models, Yun-Zhu Song and colleagues introduced the Fréchet Story Distance (FSD) in Character-Preserving Coherent Story Visualization. Unlike standard metrics that look at image quality, FSD specifically measures temporal and character consistency, penalizing models that fail to maintain a unified visual identity.

Why It Works: The Shift to Global Optimization

The success of these diverse approaches—from Sixiao Zheng's non-autoregressive ContextualStory to Yu Cheng's ReCo-RL—comes from a shared shift in perspective. Rather than optimizing for the "next best word" or "next best frame" in isolation, these models optimize for the entire trajectory.

By treating a sequence as a single, interconnected unit—whether through hierarchical planning, structured graphs, or global attention—the field is moving toward AI that doesn't just see the world, but understands the stories that connect its moments.

Papers in this version

Showing 60 of 128 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 90 papers
  2. Version 1 · 2026-10-04 · 128 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_narrative_coherence_0bb429b0,
  title = {sequential narrative coherence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-narrative-coherence/rdr_0bb429b0652b4b568c91ae016e2a6920},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_global_narrative_coherence_for_sequentia_0bb429b0_v1,
  title = {Global Narrative Coherence for Sequential Visual Storytelling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-narrative-coherence-for-sequential-visual-storytelling/rdr_0bb429b0652b4b568c91ae016e2a6920/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}