RealDiffusion: Physics-Informed Attention for Multi-Character Storybook Generation

Qi Zhao, Jun Chen, Ivor Tsang, Guang Dai

TL;DR

RealDiffusion is a training-free, inference-time framework that solves the fundamental trade-off between character consistency and narrative action in multi-character storybook generation. By modeling feature evolution using physical principles—specifically insulated heat diffusion and region-aware stochasticity—it allows users to precisely balance character coherence and storytelling dynamism using a single control knob.

What Problem Does This Paper Address?

Generating a sequence of images to tell a story (storybook generation) requires maintaining character coherence (keeping characters looking identical across frames) while enabling narrative dynamism (allowing characters to move, change poses, and interact within evolving scenes).

Existing diffusion models excel at single-image generation but struggle with sequential, multi-character setups. Current training-free methods often fall into two traps: either they suffer from "attribute swapping" (where characters accidentally trade clothing, hair colors, or identities) and identity drift, or they over-stabilize the sequence, causing the story to stagnate with frozen, unchanging characters. RealDiffusion addresses this tension without requiring expensive fine-tuning or per-subject training.

Our framework tunes the balance between storytelling dynamism and robust character coherence.

Methods

RealDiffusion introduces a lightweight Physics-informed Attention mechanism injected directly into the self-attention layers of a pre-trained U-Net (like Stable Diffusion XL) during inference.

The framework operates via three core components:

  1. Dynamic Mask Generation: At each sampling step, the model extracts cross-attention maps corresponding to the subject tokens and binarizes them using Otsu's thresholding method. This localizes the intervention strictly to the characters and prevents background leakage.
  2. PhysicsOperator: Features within the subject masks are evolved using a discrete update rule inspired by partial differential equations (PDEs). It applies insulated heat diffusion to smooth out high-frequency temporal noise (suppressing attribute drift) alongside region-aware stochasticity (controlled noise injection) to preserve pose changes and scene evolution.
  3. Disentangled Attention: In the self-attention blocks, Queries ($Q$) are constructed from the physics-smoothed features to guarantee temporal coherence. Meanwhile, Keys ($K$) and Values ($V$) are built from identity-enhanced states using an ID bank injection.

A single user-controlled parameter, the Global Trade-off Controller ($\alpha$), scales the system: a low $\alpha$ prioritizes dynamic actions, while a high $\alpha$ strengthens the physical prior to lock in character identity.

Overview of RealDiffusion architecture.

Datasets

The authors evaluated RealDiffusion on a comprehensive test set consisting of 300 multi-character prompts generated by GPT-5 to simulate complex storybook scenarios.

To evaluate the unique trade-offs of this task, the paper introduces two new sequence-aware metrics:

  • Temporal Regularity ($\mathcal{R}_t$): Measures character coherence by calculating the average $L_2$ norm of the second-order differences of per-frame CLIP features (acting as a discrete Laplacian).
  • Storytelling Quality ($\mathcal{S}_t$): Holistically evaluates the balance between coherence and dynamism using a soft-min aggregator of both properties.

Key Insights and Ideas

  • Physics as an Inference-Time Regularizer: Instead of training a neural network to learn consistency, the authors elegantly treat multi-frame feature evolution as a configurable physical system, applying classical PDE principles directly to latent features.
  • Why Heat Diffusion? The authors benchmarked several physical laws, including Burgers' equation, the Wave equation, Conservation law, and Elasticity. They discovered that the heat equation is uniquely suited because it is a dissipative system that guarantees a stable smoothing process toward equilibrium. Conservative systems (like the Wave equation) introduce unstable feature oscillations, leading to visual artifacts.
  • The Power of Insulation: By establishing a "zero-flux boundary" at the edge of the dynamic mask, the heat diffusion process is completely insulated within each character's region. This prevents character attributes from bleeding into the background or blending with other characters.
Ablation on physical priors showing that heat diffusion yields the highest character coherence without visual degradation.

Key Results

  • State-of-the-Art Coherence: RealDiffusion outperformed prominent baselines (including IP-Adapter, PhotoMaker, StoryDiffusion, and ConsiStory) on temporal coherence, achieving the highest adjacent-frame consistency (CLIP-I of 0.8242) and the lowest perceptual distance (DreamSim of 0.2780).
  • Superior Balance: It achieved the top score on the newly proposed Storytelling Quality ($\mathcal{S}_t$) metric (0.2132 vs. the next-best 0.1920 from OnePromptOneStory), proving its unique ability to maintain character identity without freezing the narrative action.
  • Efficiency: These gains are achieved with minimal computational overhead, adding only a few seconds to the total generation time (20.5 seconds total) while maintaining a low VRAM footprint (17.8 GB), making it highly practical for standard hardware.
Qualitative comparison of RealDiffusion against five state-of-the-art baseline models.

Limitations and Future Work

The authors note a slight, deliberate trade-off: prioritizing character coherence results in a minor decrease in text alignment (CLIP-T) compared to some baselines. Additionally, while the framework excels at training-free storybook generation, exploring how these physics-informed priors can be extended to continuous, long-form video generation or highly complex multi-subject interactions remains an open and promising direction.

Summary updated

Abstract

While modern diffusion models excel at generating diverse single images, extending this to sequential generation reveals a fundamental challenge: balancing narrative dynamism with multi-character coherence. Existing methods often falter at this trade-off, leading to artifacts where characters lose their identity or the story stagnates. To resolve this critical tension, we introduce RealDiffusion, a unified framework designed to reconcile robust coherence with narrative dynamism. Heat diffusion serves as a dissipative prior that averages neighboring features along the sequence and removes high-frequency noise within the subject region. This suppresses attribute drift and stabilizes identity across frames. A region-aware stochastic process then introduces small perturbations that explore nearby modes and prevent collapse so the story maintains pose change and scene evolution. We thus introduce a lightweight, training-free Physics-informed Attention mechanism that injects controllable physical priors into the self-attention layers during inference. By modeling feature evolution as a configurable physical system, our method regularizes spatio-temporal relationships without suppressing intentional, prompt-driven changes. Extensive experiments demonstrate that RealDiffusion achieves substantial gains in character coherence while preserving narrative dynamism, outperforming state-of-the-art approaches. Code is available at https://github.com/ShmilyQi-CN/RealDiffusion.
BibTeX
@inproceedings{zhao2026realdiffusion,
  author = {Qi Zhao and Jun Chen and Ivor Tsang and Guang Dai},
  title = {{RealDiffusion: Physics-Informed Attention for Multi-Character Storybook Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=p8SNKyYuq7},
}
Figure