RealDiffusion: Physics-informed Attention for Multi-character Storybook Generation

Qi Zhao, Jun Chen, Ivor Tsang, Guang Dai

TL;DR

RealDiffusion introduces a training-free, physics-informed attention mechanism that models multi-frame feature evolution as a controllable physical system. By combining an insulated heat diffusion kernel for character coherence with a region-aware stochastic process for narrative dynamism, it allows users to explicitly balance character consistency and story progression using a single control knob.

What Problem Does This Paper Address?

Generating sequential images for a storybook requires keeping multiple characters visually consistent across frames (coherence) while allowing them to move, change poses, and progress through a plot (dynamism). Current diffusion models struggle with this balance. Training-free attention-sharing methods often suffer from "attribute swapping" (e.g., characters trading clothes, hair colors, or identities mid-story) or result in static, stagnant scenes where characters barely move. RealDiffusion tackles this fundamental tension at inference time without requiring expensive fine-tuning, optimization, or per-subject training.

Methods

RealDiffusion operates entirely at inference time by modifying the self-attention layers of a pre-trained U-Net (such as Stable Diffusion XL). It views multi-frame generation as a controllable dynamical system and introduces a Physics-informed Attention mechanism guided by three core components:

  1. Dynamic Mask Generation: At each sampling step, cross-attention maps corresponding to the subject tokens are captured, averaged, and binarized using Otsu's thresholding method. This automatically creates binary masks for the subjects without any training, preventing background leakage.
  2. PhysicsOperator: This operator applies a discrete update rule based on partial differential equations (PDEs). It uses an insulated heat diffusion kernel to smooth features within the subject region across the frame sequence (suppressing high-frequency noise and attribute drift) alongside a region-aware stochastic process that injects controlled noise to preserve scene evolution and pose changes.
  3. Global Trade-off Controller ($\alpha$): A user-adjustable parameter $\alpha \in [0,1]$ that modulates the system. A high $\alpha$ strengthens heat diffusion to prioritize robust visual fidelity and coherence, while a low $\alpha$ prioritizes dynamic poses and actions.

The framework integrates these components into attention via a disentangled intervention: Queries ($Q$) are built from the physics-smoothed states to ensure temporal coherence, while Keys ($K$) and Values ($V$) are built from identity-enhanced states using an ID bank injection.

Overview of the RealDiffusion framework architecture

Datasets

Because RealDiffusion is entirely training-free, no large-scale training datasets were required. For evaluation, the authors curated a comprehensive test set consisting of 300 multi-character prompts generated by GPT-5, covering complex multi-character interactions and diverse artistic styles.

Key Insights and Ideas

  • Thermodynamics as a Regularizer: Instead of training a network to learn consistency, the authors use the mathematical principles of physical systems—specifically heat diffusion—as a direct computational tool during inference. Heat diffusion naturally acts as a dissipative, stabilizing force that averages neighboring features along a sequence to eliminate identity drift.
  • The Necessity of Insulation: Standard heat diffusion would blur the entire image sequence, causing backgrounds to bleed into characters. By introducing an "insulated Laplacian" bounded by dynamic masks, the smoothing effect is strictly confined to the characters, leaving the background free to evolve dynamically.
  • The Coherence-Dynamism Spectrum: Coherence and dynamism are inherently adversarial. RealDiffusion formalizes this by mapping them to opposing physical forces (dissipation vs. stochastic perturbation) governed by a single intuitive knob ($\alpha$), filling a gap left by prior training-free methods that lack explicit user control.

Key Results

  • Quantitative Evaluation: RealDiffusion achieved state-of-the-art performance on a newly proposed metric, Storytelling Quality ($S_t = 0.2132$), which holistically measures the balance between coherence and dynamism. It also significantly outperformed baselines in temporal smoothness metrics like CLIP-I ($0.8242$ vs. ConsiStory's $0.7905$) and DreamSim ($0.2780$, where lower is better).
  • Qualitative Superiority: Compared to state-of-the-art baselines like IP-Adapter, PhotoMaker, and StoryDiffusion, RealDiffusion successfully prevents identity drift and attribute swapping in complex multi-character interactions while maintaining vivid pose changes and scene transitions.
Qualitative comparison of RealDiffusion against five state-of-the-art baseline models

Limitations and Future Work

  • Extreme Controller Values: The authors note that the practical implications of the $\alpha$ controller mean that a very high $\alpha$ ($\alpha \rightarrow 1.0$) can lead to static characters lacking narrative dynamism, while a very low $\alpha$ diminishes the physical prior's influence, causing a loss of character coherence.
  • Text Alignment Trade-off: RealDiffusion makes a deliberate trade-off, accepting a slight decrease in text alignment (CLIP-T) compared to some baselines in exchange for vastly superior multi-character consistency and temporal coherence.
  • Future Directions: Promising future work includes extending this training-free, physics-informed inference paradigm to long-form video generation and more complex, multi-subject narrative structures.

Summary updated

Abstract

While modern diffusion models excel at generating diverse single images, extending this to sequential generation reveals a fundamental challenge: balancing narrative dynamism with multi-character coherence. Existing methods often falter at this trade-off, leading to artifacts where characters lose their identity or the story stagnates. To resolve this critical tension, we introduce RealDiffusion, a unified framework designed to reconcile robust coherence with narrative dynamism. Heat diffusion serves as a dissipative prior that averages neighboring features along the sequence and removes high-frequency noise within the subject region. This suppresses attribute drift and stabilizes identity across frames. A region-aware stochastic process then introduces small perturbations that explore nearby modes and prevent collapse so the story maintains pose change and scene evolution. We thus introduce a lightweight, training-free Physics-informed Attention mechanism that injects controllable physical priors into the self-attention layers during inference. By modeling feature evolution as a configurable physical system, our method regularizes spatio-temporal relationships without suppressing intentional, prompt-driven changes. Extensive experiments demonstrate that RealDiffusion achieves substantial gains in character coherence while preserving narrative dynamism, outperforming state-of-the-art approaches. Code is available at https://github.com/ShmilyQi-CN/RealDiffusion.
BibTeX
@inproceedings{zhao2026realdiffusion,
  author = {Qi Zhao and Jun Chen and Ivor Tsang and Guang Dai},
  title = {{RealDiffusion: Physics-informed Attention for Multi-character Storybook Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=p8SNKyYuq7},
}
Figure