VibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender

Yuzhe Chen, Muxi Diao, Lele Yang, Wenbin Xu, Xuyang Liu, Qile Su, Changyi Liu, Jiankang Chen, Kaiyu Jiang, Haoyu Zhang, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Bingyi Li, Xu Wang, Xuanyu Zheng, Bin Wen, Tingting Gao, Wentao Zhang, Jiale CaoarXiv 2026 · arXiv

TL;DR

VibePrevisBench is a new benchmark designed to evaluate how well AI agents can write Blender code to reconstruct complex, multi-shot 4D scenes (3D scenes over time) from creative briefs. The study reveals that while frontier models are competent at basic staging, they struggle with spatial reasoning, default to "common" camera choices, and—most surprisingly—fail to correct their own errors even when they "inspect" their renders.

What Problem Does This Paper Address?

Modern "controllable" video generation models often rely on previsualization (previs)—a rough, untextured 3D "graybox" video that defines the layout, camera movement, and character timing. While these proxy videos provide excellent control, creating them manually in software like Blender is a high-skill, time-consuming task.

Enter agentic coding: using LLMs to write the Python scripts that build these 3D scenes. However, existing benchmarks for 3D/4D generation are often too simple, focusing on single objects or short clips without people. They also typically evaluate only the final pixels, ignoring the underlying 3D scene structure. VibePrevisBench fills this gap by requiring agents to build long (median 40s), multi-shot scenes with multiple actors, evaluated against both the raw 3D data and the visual recognizability of the output.

The pipeline: from a brief to a 4D scene and finally a real-world video.

Methods

The authors propose a task where an agent receives a "brief" (which could be a script, a photo, a video clip to continue, or a reference video to rebuild) and must output a standalone Blender Python script.

The evaluation is two-pronged:

  1. Rule-based Scene Check: The system re-runs the agent's code in a sandbox, exports the 4D scene data (bounding boxes, camera paths, etc.), and compares it against hidden ground-truth (GT) facts across nine dimensions, such as "did the character turn at 14.0 seconds?" or "is the camera at eye level?"
  2. VLM-based Screen Check: A Vision-Language Model (Gemini 1.5 Pro) acts as a "director" judging the rendered proxy video. It performs a "blind recap" to see if it can identify characters and actions without knowing the prompt, then checks if the specific requirements of the brief are actually visible and recognizable on screen.
Comparison of Rule-based vs. VLM-based checks.

Datasets

The authors introduced VibePrevisBench, which consists of 242 tasks spanning seven domains (e.g., Homes, Offices, Nature, Sports).

  • Input Kinds: 94 scripts, 39 photo sets, 23 continuations, and 86 reference videos.
  • Complexity: 88% of tasks involve two or more shots; 79% involve two or more people.
  • Scale: The benchmark includes 99,000 individual rule checks.
Overview of the 242 tasks across domains and input types.

Key Insights and Ideas

  • The Multimodal Bottleneck: The performance gap between the best and worst agents is much wider on visual inputs (photos/videos) than on text-only scripts. This suggests that the primary hurdle isn't writing Blender code, but rather understanding space and motion from pixels.
  • The "Common Choice" Bias: Agents tend to default to "safe" choices. If a reference video uses a challenging low-angle shot, agents often revert to a standard eye-level shot. They struggle with specific camera moves like orbits, often turning them into simple sideways pans.
  • The Self-Correction Paradox: Agents are programmed to "look" at their renders (viewing 13–40 images per run), but more looking doesn't lead to better scores. In over half of the runs where agents claimed their render "looked right," they still had significant errors in camera angles or missing events. They lack the visual "taste" or discriminative power to spot their own mistakes.
  • Rules vs. Pixels: About 31% of actions that pass the "Rule-based" check (meaning the code technically performed the action) are judged as invisible or unrecognizable by the VLM. This highlights why evaluating the 3D scene alone is insufficient.

Key Results

The researchers evaluated 11 frontier agents, including models from the GPT, Gemini, Claude, and DeepSeek families.

  • Top Performer: GPT-6 Astra led with an overall score of 77.7, followed closely by GPT-6.1 Sol (76.1) and Gemini 4 Argon (75.0).
  • Efficiency: GPT-6.1 Sol emerged as a highly efficient choice, reaching 98% of the leader's score at only 21% of the API cost.
  • Failure Modes: Even the best models failed to solve the benchmark completely. Only 35 out of 242 tasks saw any agent reach a Rule-based score of 90 or higher.
The evaluation pipeline for a single video task.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Physical Validity: The current rule-based check for "physical validity" (e.g., characters floating or clipping through floors) is relatively lenient and misses subtle errors.
  • Judge Bias: While the VLM judge (Gemini) was found to be generally robust, it showed a slight (1.3 point) preference for its own model family.
  • Future Directions: The results suggest that future research should focus on improving agents' spatial "common sense" and developing more explicit, item-by-item verification loops where the agent checks every requirement of the brief against its own render before finalizing the code.

Summary updated

Abstract

Controllable real-world video generation models can condition on a rendered graybox proxy video, which fixes layout, motion and camera. Coding agents can build such videos by reconstructing a 4D scene in Blender, yet no benchmark checks both these videos and the long, multi-shot scenes behind them. We introduce VibePrevisBench, a benchmark for 4D scene reconstruction: an agent turns a brief into an executable 4D scene and renders its proxy video. Its 242 tasks span four input kinds, including scripts, photos, continuations, and reference videos. Each delivery is scored along two evaluation dimensions. Rule-based Scene Check re-runs the code and compares the exported 4D scene with hidden ground truth facts in nine dimensions. VLM-based Screen Check asks a VLM judge whether the facts that pass can be recognized in the proxy video. We analyze the evaluation results and trajectories of 11 frontier coding agents and identify three key observations. First, multimodal understanding separates the agents, especially in understanding space and camera motion. Second, all agents seem to already master basic camera work and staging, and fail at challenging requests. Third, agents inspect their own renders but fail to correct their errors. Agents thus need better visual understanding of space and camera, explicit checks for challenging requests, and item-by-item verification of renders against the brief.
BibTeX
@misc{chen2026vibeprevisbench,
  author = {Yuzhe Chen and Muxi Diao and Lele Yang and Wenbin Xu and Xuyang Liu and Qile Su and Changyi Liu and Jiankang Chen and Kaiyu Jiang and Haoyu Zhang and Hongyi Fu and Jianxiong Wang and Tianke Zhang and Haonan Fan and Yingxin Li and Bingyi Li and Xu Wang and Xuanyu Zheng and Bin Wen and Tingting Gao and Wentao Zhang and Jiale Cao},
  title = {{VibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender}},
  year = {2026},
  eprint = {2611.15789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15789},
}
Figure