IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation

Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang

Abstract

Diffusion-based models have revolutionized video generation, yet they often struggle with precise lighting control and temporal consistency due to a lack of explicit geometric integration. We present IllumiCraft, a unified diffusion framework designed for controllable video relighting. By fusing high-dynamic-range (HDR) illumination maps, 3D point tracks, and synthetic appearance cues within a Transformer-based architecture, IllumiCraft achieves superior physical plausibility and visual quality. Our approach significantly reduces temporal flickering and improves the realism of light-object interactions, outperforming existing methods in both fidelity and efficiency.

Introduction

While text-to-video generation has reached impressive milestones, video relighting—the process of altering scene illumination while maintaining temporal motion and subject identity—remains a significant challenge. Traditional methods often suffer from two primary defects: temporal flickering, where light sources shift erratically between frames, and flat interactions, where lighting fails to realistically wrap around 3D volumes or cast accurate shadows during movement.

Existing diffusion-based video editors typically treat lighting as an implicit 2D texture effect. IllumiCraft addresses these limitations by explicitly incorporating 3D geometric trajectories and HDR illumination cues into the generation process, ensuring that lighting remains consistent with the scene's underlying structure.

Methodology

IllumiCraft is built upon the Wan2.1 Diffusion Transformer (DiT) architecture. The framework integrates three distinct streams of information to guide the video synthesis:

1. Geometry Injection

To provide the model with an understanding of 3D space, IllumiCraft utilizes a "Condition DiT." This component is a lightweight clone of the initial blocks of the main model that processes 3D point tracks. By injecting these trajectories, the model learns the spatial relationship between surface normals and light directions, allowing shadows and highlights to move correctly as objects rotate or translate.

2. Illumination Tokens

Lighting environments are represented using HDR environment maps. These maps are processed through a specialized MLP-Transformer encoder into "illumination tokens." These tokens are concatenated with standard text embeddings, enabling the model to interpret the lighting environment with the same flexibility as a textual prompt.

3. Latent Fusion and Appearance Cues

The model operates in a compressed latent space. To maintain the identity of the subject during relighting, the noisy video latents are concatenated with "appearance cues"—perturbed versions of the original video frames. This ensures that while the lighting changes, the fundamental characteristics of the objects and background remain stable.

The IllumiPipe Dataset

A major contribution of this work is the IllumiPipe Dataset, a large-scale benchmark created to train and evaluate video relighting models.

  • Scale: 20,170 high-quality video pairs curated from Pexels.
  • Features: Each pair includes the original video, a relit version, synchronized HDR maps, and dense 3D tracking data.
  • Innovation: To stabilize the lighting reference, the authors developed a "chrome ball" warping technique. By estimating camera motion and warping the HDR map from the first frame onto subsequent frames, they eliminated the jitter common in single-image HDR estimators.

Results and Evaluation

IllumiCraft establishes a new state-of-the-art in controllable video generation:

  • Quantitative Performance: In text-conditioned relighting tasks, IllumiCraft achieved a 43% reduction in Fréchet Video Distance (FVD) compared to Light-A-Video, demonstrating significantly higher visual realism.
  • Efficiency: The model is highly optimized for production workflows, completing inference for a 49-frame video in 105 seconds—approximately 6x faster than AnyV2V.
  • Qualitative Quality: Visual analysis shows that IllumiCraft excels at preserving fine textures (such as fur or fabric) and handling complex scenarios like volumetric fog or multi-colored spotlights without the blurring artifacts seen in previous models.

Conclusion and Future Work

IllumiCraft demonstrates that explicit geometric guidance is the key to unlocking high-fidelity video relighting. By treating geometry as an "anchor" for light, the framework achieves a level of temporal and physical consistency previously unattainable in diffusion models.

Future research will focus on improving the handling of complex occlusions—such as objects moving between light sources—and expanding the dataset to include more extreme lighting conditions to further refine the system's robustness for professional cinematography.

Summary updated

Abstract

Although diffusion-based models can generate high-quality and high-resolution video sequences from textual or image inputs, they lack explicit integration of geometric cues when controlling scene lighting and visual appearance across frames. To address this limitation, we propose IllumiCraft, an end-to-end diffusion framework accepting three complementary inputs: (1) high-dynamic-range (HDR) video maps for detailed lighting control; (2) synthetically relit frames with randomized illumination changes (optionally paired with a static background reference image) to provide appearance cues; and (3) 3D point tracks that capture precise 3D geometry information. By integrating the lighting, appearance, and geometry cues within a unified diffusion architecture, IllumiCraft generates temporally coherent videos aligned with user-defined prompts. It supports background-conditioned and text-conditioned video relighting and provides better fidelity than existing controllable video generation methods. Project Page: https://yuanze-lin.me/IllumiCraft_page
BibTeX
@inproceedings{lin2025illumicraft,
  author = {Yuanze Lin and Yi-Wen Chen and Yi-Hsuan Tsai and Ronald Clark and Ming-Hsuan Yang},
  title = {{IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=vmYcNhs8Av},
}
Figure