TL;DR
BlenderFORGE is a new framework designed to train open-weight Multimodal Large Language Models (MLLMs) to become expert 3D graphics editors. By combining a scalable, perturbation-based data pipeline with a deterministic Blender sandbox, the authors successfully fine-tuned models like Qwen3.5-9B to generate precise Blender Python (bpy) code, achieving performance competitive with proprietary giants like GPT-5.2.
What Problem Does This Paper Address?
Professional 3D content creation tools like Blender are incredibly powerful but have a notoriously steep learning curve. While MLLMs offer a promising way to democratize 3D editing—allowing users to describe changes in natural language—training them to do so reliably is difficult.
Current MLLM-based 3D editing systems typically rely on "inference-time search," where a model tries multiple edits and uses a verifier to pick the best one. This is slow, expensive, and often relies on closed-source APIs. The field lacks two critical things for proper training:
- Scalable, high-quality training data that pairs visual "before and after" scenes with the exact code needed to transform them.
- A stable, verifiable execution environment where an agent can interact with a 3D scene, receive feedback, and be rewarded for correct programmatic edits.
Methods
The authors introduce BlenderFORGE, a closed-loop framework that integrates three main components:
- Perturbation-Based Data Generation: Instead of manually creating tasks, the pipeline takes existing 3D assets and applies random, rule-based perturbations (e.g., moving a chair, changing a light's color). This automatically creates a "start" scene and a "goal" scene, providing the ground-truth code needed for the transformation.
- Isolated Blender Sandbox: To enable Reinforcement Learning (RL), the authors built a parallelized, instance-isolated environment. It exposes a unified API for executing
bpycode, rendering previews, and exporting structured scene states (like object coordinates). This ensures that the agent’s training is deterministic and reproducible. - Two-Stage Training Pipeline:
- Stage 1: Trajectory SFT: The model is first fine-tuned using Supervised Fine-Tuning on high-quality editing traces generated by a "teacher" model (GPT-5.2).
- Stage 2: AgentLoop-RL: The model then undergoes RL (using Group Sequence Policy Optimization) within the sandbox. It receives rewards based on how much it reduces the discrepancy between the current scene and the goal scene across visual, geometric, and state-based metrics.
Datasets
The framework leverages several large-scale 3D repositories to generate its training and evaluation tasks:
- 3D-FRONT & 3D-FUTURE: Used for object placement tasks in indoor environments.
- Infinigen: Used for lighting adjustment and complex material modification tasks.
- Curated Assets: 54 high-quality assets were curated specifically for shape-key editing (fine-grained geometry changes).
- BlenderGym: Used as an independent benchmark to test the model's ability to transfer its skills to unseen tasks.
Key Insights and Ideas
- Reactive Editing is Key: Unlike "one-shot" generation, BlenderFORGE treats editing as a multi-turn dialogue. The agent can "look" at its progress via the sandbox's preview tool and adjust its code in the next turn.
- State over Pixels: While visual metrics (like CLIP distance) are useful, they can be noisy. BlenderFORGE uses "State Verifiability"—directly checking the scene graph for object positions or material values—to provide much more stable reward signals for RL.
- Amortizing Search: By training on these trajectories, the "experience" of searching for the right edit is baked into the model's weights. This makes the final agent much faster and more efficient at deployment than systems that have to search from scratch every time.
Key Results
The researchers trained two primary agents: Qwen3-VL-8B-BlenderFORGE and Qwen3.5-9B-BlenderFORGE.
- Massive Gains over Base Models: On the object placement task, the success rate ([email protected]) for the 9B model jumped from a measly 3.00% (base) to 40.00% after training.
- Competitive with Proprietary Models: The trained open-weight agents outperformed several zero-shot proprietary models and ranked just behind GPT-5.2 and Claude-Sonnet-4.6 on placement and lighting metrics.
- Generalization: The models showed strong transfer performance on the independent BlenderGym benchmark and even on "held-out" tasks like camera reframing, which were not part of the original training distribution.
Limitations and Future Work
- Task Complexity: The current framework focuses on four specific task families. It does not yet handle more complex operations like object insertion/deletion, physics-aware editing, or long-horizon scene redesign.
- Camera Control: While the agents improved, general-purpose camera navigation remains a challenge, with a "valid rate" of only 35% in held-out tests.
- Environment Specificity: The agents are currently "Blender-native." Transferring these skills to other engines (like Unreal or Unity) or CAD systems remains an open research question.
Summary updated