TL;DR
RealDiffusion is a training-free, inference-time framework that solves the fundamental trade-off between character consistency and narrative action in multi-character storybook generation. By modeling feature evolution using physical principles—specifically insulated heat diffusion and region-aware stochasticity—it allows users to precisely balance character coherence and storytelling dynamism using a single control knob.
What Problem Does This Paper Address?
Generating a sequence of images to tell a story (storybook generation) requires maintaining character coherence (keeping characters looking identical across frames) while enabling narrative dynamism (allowing characters to move, change poses, and interact within evolving scenes).
Existing diffusion models excel at single-image generation but struggle with sequential, multi-character setups. Current training-free methods often fall into two traps: either they suffer from "attribute swapping" (where characters accidentally trade clothing, hair colors, or identities) and identity drift, or they over-stabilize the sequence, causing the story to stagnate with frozen, unchanging characters. RealDiffusion addresses this tension without requiring expensive fine-tuning or per-subject training.
Methods
RealDiffusion introduces a lightweight Physics-informed Attention mechanism injected directly into the self-attention layers of a pre-trained U-Net (like Stable Diffusion XL) during inference.
The framework operates via three core components:
- Dynamic Mask Generation: At each sampling step, the model extracts cross-attention maps corresponding to the subject tokens and binarizes them using Otsu's thresholding method. This localizes the intervention strictly to the characters and prevents background leakage.
- PhysicsOperator: Features within the subject masks are evolved using a discrete update rule inspired by partial differential equations (PDEs). It applies insulated heat diffusion to smooth out high-frequency temporal noise (suppressing attribute drift) alongside region-aware stochasticity (controlled noise injection) to preserve pose changes and scene evolution.
- Disentangled Attention: In the self-attention blocks, Queries ($Q$) are constructed from the physics-smoothed features to guarantee temporal coherence. Meanwhile, Keys ($K$) and Values ($V$) are built from identity-enhanced states using an ID bank injection.
A single user-controlled parameter, the Global Trade-off Controller ($\alpha$), scales the system: a low $\alpha$ prioritizes dynamic actions, while a high $\alpha$ strengthens the physical prior to lock in character identity.
Datasets
The authors evaluated RealDiffusion on a comprehensive test set consisting of 300 multi-character prompts generated by GPT-5 to simulate complex storybook scenarios.
To evaluate the unique trade-offs of this task, the paper introduces two new sequence-aware metrics:
- Temporal Regularity ($\mathcal{R}_t$): Measures character coherence by calculating the average $L_2$ norm of the second-order differences of per-frame CLIP features (acting as a discrete Laplacian).
- Storytelling Quality ($\mathcal{S}_t$): Holistically evaluates the balance between coherence and dynamism using a soft-min aggregator of both properties.
Key Insights and Ideas
- Physics as an Inference-Time Regularizer: Instead of training a neural network to learn consistency, the authors elegantly treat multi-frame feature evolution as a configurable physical system, applying classical PDE principles directly to latent features.
- Why Heat Diffusion? The authors benchmarked several physical laws, including Burgers' equation, the Wave equation, Conservation law, and Elasticity. They discovered that the heat equation is uniquely suited because it is a dissipative system that guarantees a stable smoothing process toward equilibrium. Conservative systems (like the Wave equation) introduce unstable feature oscillations, leading to visual artifacts.
- The Power of Insulation: By establishing a "zero-flux boundary" at the edge of the dynamic mask, the heat diffusion process is completely insulated within each character's region. This prevents character attributes from bleeding into the background or blending with other characters.
Key Results
- State-of-the-Art Coherence: RealDiffusion outperformed prominent baselines (including IP-Adapter, PhotoMaker, StoryDiffusion, and ConsiStory) on temporal coherence, achieving the highest adjacent-frame consistency (CLIP-I of
0.8242) and the lowest perceptual distance (DreamSim of0.2780). - Superior Balance: It achieved the top score on the newly proposed Storytelling Quality ($\mathcal{S}_t$) metric (
0.2132vs. the next-best0.1920from OnePromptOneStory), proving its unique ability to maintain character identity without freezing the narrative action. - Efficiency: These gains are achieved with minimal computational overhead, adding only a few seconds to the total generation time (
20.5seconds total) while maintaining a low VRAM footprint (17.8GB), making it highly practical for standard hardware.
Limitations and Future Work
The authors note a slight, deliberate trade-off: prioritizing character coherence results in a minor decrease in text alignment (CLIP-T) compared to some baselines. Additionally, while the framework excels at training-free storybook generation, exploring how these physics-informed priors can be extended to continuous, long-form video generation or highly complex multi-subject interactions remains an open and promising direction.
Summary updated