TL;DR
The authors introduce DAR (Diffusion as Renderer), a novel approach that transforms pretrained video diffusion models into precise 4D renderers guided by an animated mesh, a camera trajectory, and a reference image. By replacing standard camera-dependent depth maps with a camera-invariant neural 4D G-buffer (tracking + world position + normals), DAR successfully decouples observer motion from object animation, setting a new state-of-the-art for controllable video generation.
What Problem Does This Paper Address?
While generative video models have become incredibly powerful, using them as controllable rendering engines for 3D/4D graphics pipelines remains a challenge. Traditional video generation models inherently entangle observer (camera) motion and scene-internal (object) motion.
Existing control mechanisms fall short: camera-control models (using Plücker rays) ignore foreground object animation, while depth-conditioned models use view-dependent depth maps that mix camera and object movements together. This makes it difficult for a model to figure out which moving surface point it is looking at, leading to visual artifacts, hallucinations, or a failure to adhere to the specified geometry and style.
Methods
DAR builds on top of the pretrained Wan2.2-Fun-5B-Control-Camera backbone. The core architectural modification is widening the model's early control adapter from 24 channels (originally just for Plücker rays) to 33 channels to accommodate a joint camera-plus-geometry interface.
Instead of standard depth maps, DAR conditions the video model on a neural 4D G-buffer projected from the animated mesh, which consists of 9 channels:
- Tracking (3 channels): Assigns a persistent color/identity to surface elements, telling the model what object or part it is tracking across frames to ensure consistent appearance transport.
- World Position (3 channels): Provides scene-normalized 3D coordinates, telling the model where that surface element is in the world, completely independent of the camera's viewpoint.
- Normal Maps (3 channels): Supplies local surface orientation to assist with shading and silhouette details.
The model is trained using either parameter-efficient LoRA fine-tuning or a full fine-tune, optimizing a standard flow-matching objective.
Datasets
To train and evaluate DAR, the authors introduced DAR-4D, a comprehensive, renderer-ready corpus compiled from diverse 3D/4D sources including BlenderKit scenes, Unreal Engine environments, and rigged Z-OO animal assets.
The dataset provides paired RGB videos, calibrated cameras, Plücker rays, and per-frame geometric buffers (depth, normals, world position, and tracking). The primary quantitative evaluations are conducted on the 4d_vis benchmark (68 clips), while generalization is tested on the OOD-34 split containing completely unseen scenes and trajectories.
Key Insights and Ideas
- Depth is a Camera Chart; World Position is a Scene Chart: The fundamental "aha!" moment of the paper is that depth is inherently camera-dependent. If a camera translates, the depth values of a static object change, forcing the video model to mentally compute inverse projections. World position, however, isolates object motion from camera motion.
- The Power of the (Tracking + World Position) Pair: Tracking handles appearance transport (maintaining identity during occlusions or motion), while world position handles geometric state (where things are). Together, they form a camera-invariant 4D state code that perfectly factorizes the rendering problem when paired with Plücker rays.
- Partial Decoupling of Style and Geometry: Because the geometry is explicitly enforced via the 4D G-buffer, the first-frame reference image can be swapped to completely alter the paint style, color blocks, or material of the scene without altering the underlying animation or camera path.
Key Results
DAR significantly outperforms both off-the-shelf video models and specialized baselines on the 4d_vis benchmark:
- Quantitative Performance: LoRA-trained DAR achieves a PSNR of 23.22, an SSIM of 0.895, and an LPIPS of 0.134, outperforming the off-the-shelf Wan2.2-Depth baseline by +1.54 dB PSNR. A full fine-tune pushes this further to 25.36 PSNR (+3.68 dB over Wan2.2-Depth).
- The Depth-Swap Ablation: To prove their geometric hypothesis, the authors ran an ablation replacing world position with depth while keeping everything else identical. Replacing world position with depth caused a consistent drop of 1.26 to 1.55 dB PSNR across all training checkpoints, validating that world position is a superior conditioning signal.
- Qualitative Fidelity: As shown in the rotation galleries, DAR accurately tracks complex animations (like flapping wings or articulated limb motions) even under extreme $\pm60^\circ$ camera orbits where baseline models typically hallucinate or lose track of the object.
Limitations and Future Work
Despite its strong performance, the authors note several limitations:
- Inference Speed: DAR is not real-time. Generating an 81-frame video at $480\times832$ resolution takes roughly 1.5 to 2.5 minutes on an enterprise GPU (A100/H100).
- Long-Video Flickering: In videos extending beyond 200 frames, visual flickering can occur under drastic camera changes or prolonged occlusions.
- Open-Category Generalization: Performance drops on the out-of-distribution (
OOD-34) dataset, indicating that the LoRA capacity or training data diversity still needs expansion for open-world deployment. - Input Dependency: The model assumes clean, accurate 4D mesh inputs; adapting it to noisy, real-world reconstructions will require integrating confidence masks or robustness layers.
Summary updated