Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh

Junhao Chen, Mingjin Chen, Henghaofan Zhang, Minglin Chen, Liaoyuan Fan, Boran Zhang, Saining Zhang, Mingze Sun, Hao Zhao, Ruqi Huang, Zhihao Li, Yufei WangarXiv 2026 · arXiv

TL;DR

The authors introduce DAR (Diffusion as Renderer), a novel approach that transforms pretrained video diffusion models into precise 4D renderers guided by an animated mesh, a camera trajectory, and a reference image. By replacing standard camera-dependent depth maps with a camera-invariant neural 4D G-buffer (tracking + world position + normals), DAR successfully decouples observer motion from object animation, setting a new state-of-the-art for controllable video generation.

What Problem Does This Paper Address?

While generative video models have become incredibly powerful, using them as controllable rendering engines for 3D/4D graphics pipelines remains a challenge. Traditional video generation models inherently entangle observer (camera) motion and scene-internal (object) motion.

Existing control mechanisms fall short: camera-control models (using Plücker rays) ignore foreground object animation, while depth-conditioned models use view-dependent depth maps that mix camera and object movements together. This makes it difficult for a model to figure out which moving surface point it is looking at, leading to visual artifacts, hallucinations, or a failure to adhere to the specified geometry and style.

Methods

DAR builds on top of the pretrained Wan2.2-Fun-5B-Control-Camera backbone. The core architectural modification is widening the model's early control adapter from 24 channels (originally just for Plücker rays) to 33 channels to accommodate a joint camera-plus-geometry interface.

Instead of standard depth maps, DAR conditions the video model on a neural 4D G-buffer projected from the animated mesh, which consists of 9 channels:

  1. Tracking (3 channels): Assigns a persistent color/identity to surface elements, telling the model what object or part it is tracking across frames to ensure consistent appearance transport.
  2. World Position (3 channels): Provides scene-normalized 3D coordinates, telling the model where that surface element is in the world, completely independent of the camera's viewpoint.
  3. Normal Maps (3 channels): Supplies local surface orientation to assist with shading and silhouette details.
Fig. 2. DAR pipeline. An animated 4D mesh, target camera trajectory, reference image, and text prompt are converted into aligned video-diffusion conditions. Plücker rays encode the observer; a mesh projection renderer produces tracking, world-position, and normal maps; a widened SimpleAdapter injects the combined camera-plus-geometry control into the pretrained Wan2.2 video DiT.

The model is trained using either parameter-efficient LoRA fine-tuning or a full fine-tune, optimizing a standard flow-matching objective.

Datasets

To train and evaluate DAR, the authors introduced DAR-4D, a comprehensive, renderer-ready corpus compiled from diverse 3D/4D sources including BlenderKit scenes, Unreal Engine environments, and rigged Z-OO animal assets.

The dataset provides paired RGB videos, calibrated cameras, Plücker rays, and per-frame geometric buffers (depth, normals, world position, and tracking). The primary quantitative evaluations are conducted on the 4d_vis benchmark (68 clips), while generalization is tested on the OOD-34 split containing completely unseen scenes and trajectories.

Key Insights and Ideas

  • Depth is a Camera Chart; World Position is a Scene Chart: The fundamental "aha!" moment of the paper is that depth is inherently camera-dependent. If a camera translates, the depth values of a static object change, forcing the video model to mentally compute inverse projections. World position, however, isolates object motion from camera motion.
  • The Power of the (Tracking + World Position) Pair: Tracking handles appearance transport (maintaining identity during occlusions or motion), while world position handles geometric state (where things are). Together, they form a camera-invariant 4D state code that perfectly factorizes the rendering problem when paired with Plücker rays.
  • Partial Decoupling of Style and Geometry: Because the geometry is explicitly enforced via the 4D G-buffer, the first-frame reference image can be swapped to completely alter the paint style, color blocks, or material of the scene without altering the underlying animation or camera path.
Fig. 1. DAR is a reference-guided 4D renderer: with the animated mesh and camera fixed, changing only the first-frame reference image re-renders the scene’s appearance. The same animated car mesh, camera trajectory, and 4D geometry are held fixed; each column uses a different reference style and each row is a normalized animation time. DAR alters paint style, color, and vehicle appearance while keeping the camera, pose, and silhouette aligned, indicating partial decoupling between geometry and appearance.

Key Results

DAR significantly outperforms both off-the-shelf video models and specialized baselines on the 4d_vis benchmark:

  • Quantitative Performance: LoRA-trained DAR achieves a PSNR of 23.22, an SSIM of 0.895, and an LPIPS of 0.134, outperforming the off-the-shelf Wan2.2-Depth baseline by +1.54 dB PSNR. A full fine-tune pushes this further to 25.36 PSNR (+3.68 dB over Wan2.2-Depth).
  • The Depth-Swap Ablation: To prove their geometric hypothesis, the authors ran an ablation replacing world position with depth while keeping everything else identical. Replacing world position with depth caused a consistent drop of 1.26 to 1.55 dB PSNR across all training checkpoints, validating that world position is a superior conditioning signal.
  • Qualitative Fidelity: As shown in the rotation galleries, DAR accurately tracks complex animations (like flapping wings or articulated limb motions) even under extreme $\pm60^\circ$ camera orbits where baseline models typically hallucinate or lose track of the object.
Qualitative comparison against baselines

Limitations and Future Work

Despite its strong performance, the authors note several limitations:

  • Inference Speed: DAR is not real-time. Generating an 81-frame video at $480\times832$ resolution takes roughly 1.5 to 2.5 minutes on an enterprise GPU (A100/H100).
  • Long-Video Flickering: In videos extending beyond 200 frames, visual flickering can occur under drastic camera changes or prolonged occlusions.
  • Open-Category Generalization: Performance drops on the out-of-distribution (OOD-34) dataset, indicating that the LoRA capacity or training data diversity still needs expansion for open-world deployment.
  • Input Dependency: The model assumes clean, accurate 4D mesh inputs; adapting it to noisy, real-world reconstructions will require integrating confidence masks or robustness layers.

Summary updated

Abstract

Pretrained video diffusion models can act as renderers when the desired scene state is already specified by an animated mesh, a camera trajectory, and a reference image. This 4D generative rendering setting raises a representation question: what image-format condition lets a video backbone obey both camera motion and scene-internal animation? We propose DAR, a reference-guided renderer that extends Wan2.2 camera control from Plücker rays alone to a joint camera-plus-geometry interface. DAR projects a neural 4D G-buffer (tracking, world position, and normal) from the animated mesh and injects it through a widened control adapter while preserving the pretrained image-to-video prior. The central design choice is the pair of tracking and world position. Tracking identifies the persistent surface element that should carry appearance; world position gives its current scene-coordinate state; normal supplies local shape. Depth plus calibrated rays can recover 3D in principle, but depth is a camera-dependent chart in which camera and object motion are mixed. On the 68-case DAR-4D benchmark, LoRA DAR reaches PSNR 23.22, SSIM 0.895, and LPIPS 0.134, improving over off-the-shelf Wan2.2-Depth by 1.54 dB PSNR; a full fine-tune reaches PSNR 25.36 and SSIM 0.917. Matched ablations show that replacing world position by depth reduces PSNR by 1.26--1.55 dB at every checkpoint, supporting tracking+world-position correspondence as a practical 4D rendering condition.
BibTeX
@misc{chen2026video,
  author = {Junhao Chen and Mingjin Chen and Henghaofan Zhang and Minglin Chen and Liaoyuan Fan and Boran Zhang and Saining Zhang and Mingze Sun and Hao Zhao and Ruqi Huang and Zhihao Li and Yufei Wang},
  title = {{Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh}},
  year = {2026},
  eprint = {2608.00094},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.00094},
}
Figure