Standard vision-language models excel at static, isolated tasks. Give a model a single image, and it can generate a highly accurate caption. Give it a single prompt, and it can synthesize a stunning visual. However, when these models are tasked with processing or generating sequences of images and text—such as photo albums, multi-panel memes, or step-by-step instructional manuals—the illusion of intelligence frequently shatters.
As Xudong Hong et al. observed in the Visually Grounded Story Generation Challenge, mastering short-form tasks like image captioning does not automatically translate to maintaining logical flow across a multi-sentence story. Instead, models treat sequential inputs as independent scenes, resulting in disjointed narratives, repetitive phrasing, and a complete loss of chronological or causal progression.
To build systems that truly understand how events unfold over time, researchers are developing architectures and training paradigms specifically designed to enforce sequential narrative coherence.
The Root of Sequential Disconnect
Why do state-of-the-art Multimodal Large Language Models (MLLMs) struggle with image streams? The core issue lies in how they handle context across boundaries. When an MLLM processes a sequence of images, it often suffers from inter-image cross-contamination or complete context omission.
As detailed by Xudong Li et al. in Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs, models frequently attribute visual details from one image to an entirely different frame in the series, or ignore specific frames altogether. This lack of inter-image coherence makes it impossible to maintain a consistent narrative thread.
This failure spans multiple domains:
- Multi-panel Compositions: In multi-image memes, each frame might represent a distinct narrative stage or emotional shift. Yuyan Chen et al. note in XMeCap: Meme Caption Generation with Sub-Image Adaptability that models fail when they treat these panels in isolation rather than integrating them into a singular, cohesive punchline.
- Interleaved Documents: When generating procedural content like recipes or DIY manuals, standard training objectives do not inherently enforce the sequential dependencies required to alternate text and images logically, a limitation highlighted by Lin Li et al. in CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation.
To bridge these semantic and temporal gaps, models must be forced to look beyond individual frames and model the transitions between them.
Blueprinting and Hierarchical Planning
One of the most effective ways to prevent a model from "losing the plot" is to separate high-level narrative planning from low-level text generation. When a model generates text autoregressively frame-by-frame, it is highly susceptible to drifting away from the central theme.
To ground this process, Danyang Liu et al. introduced a "blueprint" approach in Visual Storytelling with Question-Answer Plans. Instead of mapping images directly to a story, the framework forces the model to first generate a sequence of question-answer (QA) pairs about the image stream. This QA plan acts as an intermediate roadmap, identifying salient concepts and establishing the necessary connective tissue before any narrative text is written.
[Image Sequence] ──> [QA Blueprint Plan] ──> [Coherent Narrative Paragraph]
│
└── (Establishes logical & causal transitions)
Hierarchical architectures operationalize this intuition by splitting responsibilities between specialized layers or agents:
- Manager-Worker Frameworks: In the framework proposed by Qiuyuan Huang in Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation, a high-level "Manager" agent plans the overarching topics across the image sequence to ensure global coherence, while a low-level "Worker" agent focuses purely on executing individual sentences locally.
- Communicating Agents: Similarly, Ruize Wang et al. developed Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication, which uses a topic generator and a story generator as two distinct entities that iteratively refine each other's work, keeping the final output anchored to a stable thematic trajectory.
Modeling Visual and Textual Transitions
Another core reason for narrative breakdown is that standard vision-language alignment techniques (like CLIP) optimize for static, paired data. They lack an understanding of change—the delta between states.
To inject temporal logic into pretrained models without massive retraining, Youngjae Yu et al. proposed Transitional Adaptation of Pretrained Models for Visual Storytelling. Their framework, TAPM, introduces a self-supervised "bridge" step that applies a sequential coherence loss to unlabeled video or image streams. By training the model to predict how visual states shift from frame to frame, it harmonizes separately pretrained visual and language modules before they are fine-tuned on storytelling tasks.
This transition modeling is equally vital for the inverse task: Story Visualization (generating a sequence of images from a multi-sentence text). If a text-to-image model treats sentences as independent prompts, characters and environments will radically change from frame to frame.
To solve this, the authors of StoryGAN: A Sequential Conditional GAN for Story Visualization, Yitong Li 0001 et al., implemented a specialized Context Encoder alongside dual-level discriminators. The context encoder tracks the story's global progress, ensuring that the generation of the current frame is mathematically conditioned on the history of the entire narrative arc, preserving character and scene consistency.
Memory Mechanisms to Prevent Repetition
Autoregressive language generation naturally suffers from recency bias. When generating a long paragraph describing a complex scene or a sequence of events, a model can easily forget what it has already described, leading to severe repetition.
To combat this, Chunpu Xu et al. designed the Interactive Key-Value Memory-augmented Attention for Image Paragraph Captioning (IMAP) model. IMAP incorporates a dynamic key-value memory system that explicitly records the model's "attention history."
By tracking which visual regions have already been attended to and translated into text, the model prevents itself from re-visiting the same objects. This memory-driven approach ensures that multi-sentence descriptions remain comprehensive and logically progressive, rather than looping over the same salient visual features.
Redefining Evaluation for Sequential Flow
How do we know if a visual narrative actually works? Traditional metrics like BLEU, METEOR, or CIDEr measure token overlap against ground-truth text, while standard image metrics focus on individual frame fidelity. Neither can detect whether a sequence follows a logical, causal plot.
To address this evaluation gap, Yun-Zhu Song et al. introduced the Fréchet Story Distance (FSD) in Character-Preserving Coherent Story Visualization. Unlike video metrics that prioritize pixel-level temporal smoothness, FSD evaluates the logical and visual flow across short image sequences, capturing whether the generated sequence accurately reflects the narrative progression described in the source text.
Furthermore, the mathematical definition of what constitutes a "visual narrative" has been refined. In developing the CoMM dataset, Lin Li et al. formalized a specialized evaluation metric that balances two competing forces:
$$\text{Narrative Coherence} \propto \text{High Consecutive Similarity} \times \text{High Global Diversity}$$
This formula captures the essence of sequential coherence. For a story to flow, consecutive frames must share high semantic similarity to maintain context and avoid jarring, disconnected jumps. Yet, across the entire sequence, there must be high diversity to ensure the story actually progresses rather than stagnating in repetitive loops. By optimizing for this delicate balance, modern architectures are moving closer to generating truly human-like visual stories.