TL;DR
Coda-DaVinci is a new "Code2Video" framework that uses LLM agents to build 3D scene proxies to guide video generation. By intelligently choosing when to use 3D guidance versus direct generation and optimizing the granularity of 3D models, it achieves state-of-the-art results in instruction following and scene consistency while maintaining natural motion.
What Problem Does This Paper Address?
While modern video generation models (like Sora or Kling) produce visually stunning results, they often fail at "boring" but essential tasks: keeping a room's layout consistent when the camera turns away, following precise spatial instructions (e.g., "put the blue box exactly behind the red one"), or obeying the laws of physics.
The "Code2Video" paradigm attempts to fix this by having an AI agent write code (like Blender scripts) to create a simple 3D "proxy" of the scene. This proxy acts as a skeletal guide for the video model. However, this approach introduces a new set of questions: How detailed should the 3D proxy be? Should every movement be simulated with physics? When does a 3D guide actually make the video look worse by making motion appear stiff?
Methods
The authors introduce Coda-DaVinci, a task-adaptive agentic framework designed to navigate these trade-offs. Its architecture, shown in the figure below, moves beyond simple "one-size-fits-all" 3D guidance.
Key components include:
- Task-Adaptive Routing: A lead agent analyzes the user's request. If the task requires natural, complex body language (where 3D proxies often look "robotic"), it opts for Direct Generation. If the task requires strict spatial control or scene revisitation, it triggers 3D-Guided Generation.
- Multi-Granularity Modeling: Instead of modeling everything at the same detail level, Coda-DaVinci uses a "mixed-res" approach. It uses medium-grained proxies for environments to ensure structural consistency but switches to coarse, simplified proxies for humans. This allows the video model to "fill in" natural joint movements and skin textures that a simple 3D script might get wrong.
- Physics-Grounded Synthesis: The system selectively invokes Blender’s physics engine for interactions like collisions or fluid dynamics, ensuring the "guide" follows Newtonian logic.
- Agentic Harness & Review: A multi-agent system manages the workflow. An "Acceptance Agent" independently reviews the candidate video against the original instructions, identifying errors and triggering "repairs" in the 3D code or the generation prompt if necessary.
Datasets
The researchers evaluated Coda-DaVinci across a diverse set of benchmarks to test different capabilities:
- WorldMemBench: Tests scene consistency during camera revisits.
- VGIF-Bench: Evaluates complex spatio-temporal instruction following.
- Principia: Focuses on physical process simulation (gravity, collisions, etc.).
- VTR (Visual Text Rendering): Measures how well the model renders specific text in a scene.
- R2M: A benchmark for evaluating revisit memory and relative consistency.
- T2AV-Compass: Used to assess general video realism and instruction following.
Key Insights and Ideas
- The "Stiffness" Trap: One of the most surprising findings is that more detail isn't always better. Fine-grained 3D proxies (G2) often lead to lower visual realism because they force the video model to follow jittery or anatomically incorrect 3D joints. Coarse proxies (G0) for humans actually yield more natural motion because they provide a spatial "anchor" while letting the video model's internal priors handle the nuances of movement.
- Prompt Binding is the Secret Sauce: Simply providing a 3D depth map or wireframe isn't enough. The authors found that "PE2" (agent-guided prompt enhancement), where the agent explicitly tells the video model "the red cylinder in the proxy is a person wearing a red shirt," significantly improves subject consistency.
- Physics as a Prior: Explicitly simulating physics in the proxy (P1) consistently improved scores across all benchmarks, particularly in "physical plausibility," proving that video models still benefit heavily from external "world engine" guidance.
Key Results
Coda-DaVinci demonstrated significant leads over both direct generation and existing Code2Video baselines:
- Instruction Following: On the VTR benchmark, it reduced the Word Error Rate (WER) from 59.21% (direct generation) to just 11.08%.
- Consistency: It achieved the highest scores on R2M and WorldMemBench, successfully maintaining object states even when they were off-screen for long periods.
- Selective Modeling: As seen in Figure 2, Coda-DaVinci can model a rice cake being sliced while leaving the complex articulation of the human hands to the video generator, resulting in a more realistic interaction than modeling the hands in 3D.
Limitations and Future Work
Despite its strengths, the authors note several areas for improvement:
- Transition Stability: In very long videos, the transitions between segments can still be shaky.
- Setting Adherence: While it's great at keeping objects consistent, the overall "vibe" or background setting can sometimes drift over long sequences.
- Computational Cost: Using a full 3D engine like Blender is more "token-heavy" and computationally expensive than simpler web-based renderers like Three.js.
Future work will likely focus on better preserving explicit control during the synthesis stage to prevent the video model from "hallucinating" away from the 3D guide during complex transitions.
Summary updated