TL;DR
RealDiffusion introduces a training-free, physics-informed attention mechanism that models multi-frame feature evolution as a controllable physical system. By combining an insulated heat diffusion kernel for character coherence with a region-aware stochastic process for narrative dynamism, it allows users to explicitly balance character consistency and story progression using a single control knob.
What Problem Does This Paper Address?
Generating sequential images for a storybook requires keeping multiple characters visually consistent across frames (coherence) while allowing them to move, change poses, and progress through a plot (dynamism). Current diffusion models struggle with this balance. Training-free attention-sharing methods often suffer from "attribute swapping" (e.g., characters trading clothes, hair colors, or identities mid-story) or result in static, stagnant scenes where characters barely move. RealDiffusion tackles this fundamental tension at inference time without requiring expensive fine-tuning, optimization, or per-subject training.
Methods
RealDiffusion operates entirely at inference time by modifying the self-attention layers of a pre-trained U-Net (such as Stable Diffusion XL). It views multi-frame generation as a controllable dynamical system and introduces a Physics-informed Attention mechanism guided by three core components:
- Dynamic Mask Generation: At each sampling step, cross-attention maps corresponding to the subject tokens are captured, averaged, and binarized using Otsu's thresholding method. This automatically creates binary masks for the subjects without any training, preventing background leakage.
- PhysicsOperator: This operator applies a discrete update rule based on partial differential equations (PDEs). It uses an insulated heat diffusion kernel to smooth features within the subject region across the frame sequence (suppressing high-frequency noise and attribute drift) alongside a region-aware stochastic process that injects controlled noise to preserve scene evolution and pose changes.
- Global Trade-off Controller ($\alpha$): A user-adjustable parameter $\alpha \in [0,1]$ that modulates the system. A high $\alpha$ strengthens heat diffusion to prioritize robust visual fidelity and coherence, while a low $\alpha$ prioritizes dynamic poses and actions.
The framework integrates these components into attention via a disentangled intervention: Queries ($Q$) are built from the physics-smoothed states to ensure temporal coherence, while Keys ($K$) and Values ($V$) are built from identity-enhanced states using an ID bank injection.
Datasets
Because RealDiffusion is entirely training-free, no large-scale training datasets were required. For evaluation, the authors curated a comprehensive test set consisting of 300 multi-character prompts generated by GPT-5, covering complex multi-character interactions and diverse artistic styles.
Key Insights and Ideas
- Thermodynamics as a Regularizer: Instead of training a network to learn consistency, the authors use the mathematical principles of physical systems—specifically heat diffusion—as a direct computational tool during inference. Heat diffusion naturally acts as a dissipative, stabilizing force that averages neighboring features along a sequence to eliminate identity drift.
- The Necessity of Insulation: Standard heat diffusion would blur the entire image sequence, causing backgrounds to bleed into characters. By introducing an "insulated Laplacian" bounded by dynamic masks, the smoothing effect is strictly confined to the characters, leaving the background free to evolve dynamically.
- The Coherence-Dynamism Spectrum: Coherence and dynamism are inherently adversarial. RealDiffusion formalizes this by mapping them to opposing physical forces (dissipation vs. stochastic perturbation) governed by a single intuitive knob ($\alpha$), filling a gap left by prior training-free methods that lack explicit user control.
Key Results
- Quantitative Evaluation: RealDiffusion achieved state-of-the-art performance on a newly proposed metric, Storytelling Quality ($S_t = 0.2132$), which holistically measures the balance between coherence and dynamism. It also significantly outperformed baselines in temporal smoothness metrics like CLIP-I ($0.8242$ vs. ConsiStory's $0.7905$) and DreamSim ($0.2780$, where lower is better).
- Qualitative Superiority: Compared to state-of-the-art baselines like IP-Adapter, PhotoMaker, and StoryDiffusion, RealDiffusion successfully prevents identity drift and attribute swapping in complex multi-character interactions while maintaining vivid pose changes and scene transitions.
Limitations and Future Work
- Extreme Controller Values: The authors note that the practical implications of the $\alpha$ controller mean that a very high $\alpha$ ($\alpha \rightarrow 1.0$) can lead to static characters lacking narrative dynamism, while a very low $\alpha$ diminishes the physical prior's influence, causing a loss of character coherence.
- Text Alignment Trade-off: RealDiffusion makes a deliberate trade-off, accepting a slight decrease in text alignment (CLIP-T) compared to some baselines in exchange for vastly superior multi-character consistency and temporal coherence.
- Future Directions: Promising future work includes extending this training-free, physics-informed inference paradigm to long-form video generation and more complex, multi-subject narrative structures.
Summary updated