TL;DR
VibePrevisBench is a new benchmark designed to evaluate how well AI agents can write Blender code to reconstruct complex, multi-shot 4D scenes (3D scenes over time) from creative briefs. The study reveals that while frontier models are competent at basic staging, they struggle with spatial reasoning, default to "common" camera choices, and—most surprisingly—fail to correct their own errors even when they "inspect" their renders.
What Problem Does This Paper Address?
Modern "controllable" video generation models often rely on previsualization (previs)—a rough, untextured 3D "graybox" video that defines the layout, camera movement, and character timing. While these proxy videos provide excellent control, creating them manually in software like Blender is a high-skill, time-consuming task.
Enter agentic coding: using LLMs to write the Python scripts that build these 3D scenes. However, existing benchmarks for 3D/4D generation are often too simple, focusing on single objects or short clips without people. They also typically evaluate only the final pixels, ignoring the underlying 3D scene structure. VibePrevisBench fills this gap by requiring agents to build long (median 40s), multi-shot scenes with multiple actors, evaluated against both the raw 3D data and the visual recognizability of the output.
Methods
The authors propose a task where an agent receives a "brief" (which could be a script, a photo, a video clip to continue, or a reference video to rebuild) and must output a standalone Blender Python script.
The evaluation is two-pronged:
- Rule-based Scene Check: The system re-runs the agent's code in a sandbox, exports the 4D scene data (bounding boxes, camera paths, etc.), and compares it against hidden ground-truth (GT) facts across nine dimensions, such as "did the character turn at 14.0 seconds?" or "is the camera at eye level?"
- VLM-based Screen Check: A Vision-Language Model (Gemini 1.5 Pro) acts as a "director" judging the rendered proxy video. It performs a "blind recap" to see if it can identify characters and actions without knowing the prompt, then checks if the specific requirements of the brief are actually visible and recognizable on screen.
Datasets
The authors introduced VibePrevisBench, which consists of 242 tasks spanning seven domains (e.g., Homes, Offices, Nature, Sports).
- Input Kinds: 94 scripts, 39 photo sets, 23 continuations, and 86 reference videos.
- Complexity: 88% of tasks involve two or more shots; 79% involve two or more people.
- Scale: The benchmark includes 99,000 individual rule checks.
Key Insights and Ideas
- The Multimodal Bottleneck: The performance gap between the best and worst agents is much wider on visual inputs (photos/videos) than on text-only scripts. This suggests that the primary hurdle isn't writing Blender code, but rather understanding space and motion from pixels.
- The "Common Choice" Bias: Agents tend to default to "safe" choices. If a reference video uses a challenging low-angle shot, agents often revert to a standard eye-level shot. They struggle with specific camera moves like orbits, often turning them into simple sideways pans.
- The Self-Correction Paradox: Agents are programmed to "look" at their renders (viewing 13–40 images per run), but more looking doesn't lead to better scores. In over half of the runs where agents claimed their render "looked right," they still had significant errors in camera angles or missing events. They lack the visual "taste" or discriminative power to spot their own mistakes.
- Rules vs. Pixels: About 31% of actions that pass the "Rule-based" check (meaning the code technically performed the action) are judged as invisible or unrecognizable by the VLM. This highlights why evaluating the 3D scene alone is insufficient.
Key Results
The researchers evaluated 11 frontier agents, including models from the GPT, Gemini, Claude, and DeepSeek families.
- Top Performer: GPT-6 Astra led with an overall score of 77.7, followed closely by GPT-6.1 Sol (76.1) and Gemini 4 Argon (75.0).
- Efficiency: GPT-6.1 Sol emerged as a highly efficient choice, reaching 98% of the leader's score at only 21% of the API cost.
- Failure Modes: Even the best models failed to solve the benchmark completely. Only 35 out of 242 tasks saw any agent reach a Rule-based score of 90 or higher.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Physical Validity: The current rule-based check for "physical validity" (e.g., characters floating or clipping through floors) is relatively lenient and misses subtle errors.
- Judge Bias: While the VLM judge (Gemini) was found to be generally robust, it showed a slight (1.3 point) preference for its own model family.
- Future Directions: The results suggest that future research should focus on improving agents' spatial "common sense" and developing more explicit, item-by-item verification loops where the agent checks every requirement of the brief against its own render before finalizing the code.
Summary updated