BlenderFORGE: Framework for Optimizing Reactive 3D-Graphics Editing Ability of MLLMs

Zilin Guo, Tianrui Zhang, Yi Rong, Yichen Liu, Yuxin Guo, Jingcheng Ni, Lewei Lu, Zehuan Wu, Dan XuarXiv 2026 · arXiv

TL;DR

BlenderFORGE is a new framework designed to train open-weight Multimodal Large Language Models (MLLMs) to become expert 3D graphics editors. By combining a scalable, perturbation-based data pipeline with a deterministic Blender sandbox, the authors successfully fine-tuned models like Qwen3.5-9B to generate precise Blender Python (bpy) code, achieving performance competitive with proprietary giants like GPT-5.2.

What Problem Does This Paper Address?

Professional 3D content creation tools like Blender are incredibly powerful but have a notoriously steep learning curve. While MLLMs offer a promising way to democratize 3D editing—allowing users to describe changes in natural language—training them to do so reliably is difficult.

Current MLLM-based 3D editing systems typically rely on "inference-time search," where a model tries multiple edits and uses a verifier to pick the best one. This is slow, expensive, and often relies on closed-source APIs. The field lacks two critical things for proper training:

  1. Scalable, high-quality training data that pairs visual "before and after" scenes with the exact code needed to transform them.
  2. A stable, verifiable execution environment where an agent can interact with a 3D scene, receive feedback, and be rewarded for correct programmatic edits.

Methods

The authors introduce BlenderFORGE, a closed-loop framework that integrates three main components:

  1. Perturbation-Based Data Generation: Instead of manually creating tasks, the pipeline takes existing 3D assets and applies random, rule-based perturbations (e.g., moving a chair, changing a light's color). This automatically creates a "start" scene and a "goal" scene, providing the ground-truth code needed for the transformation.
  2. Isolated Blender Sandbox: To enable Reinforcement Learning (RL), the authors built a parallelized, instance-isolated environment. It exposes a unified API for executing bpy code, rendering previews, and exporting structured scene states (like object coordinates). This ensures that the agent’s training is deterministic and reproducible.
  3. Two-Stage Training Pipeline:
    • Stage 1: Trajectory SFT: The model is first fine-tuned using Supervised Fine-Tuning on high-quality editing traces generated by a "teacher" model (GPT-5.2).
  4. Stage 2: AgentLoop-RL: The model then undergoes RL (using Group Sequence Policy Optimization) within the sandbox. It receives rewards based on how much it reduces the discrepancy between the current scene and the goal scene across visual, geometric, and state-based metrics.
Figure 1: Overview of the learning process under the BlenderFORGE framework. MLLMs learn the correspondence between visual changes and 3D-graphics edits from scene perturbations.

Datasets

The framework leverages several large-scale 3D repositories to generate its training and evaluation tasks:

  • 3D-FRONT & 3D-FUTURE: Used for object placement tasks in indoor environments.
  • Infinigen: Used for lighting adjustment and complex material modification tasks.
  • Curated Assets: 54 high-quality assets were curated specifically for shape-key editing (fine-grained geometry changes).
  • BlenderGym: Used as an independent benchmark to test the model's ability to transfer its skills to unseen tasks.

Key Insights and Ideas

  • Reactive Editing is Key: Unlike "one-shot" generation, BlenderFORGE treats editing as a multi-turn dialogue. The agent can "look" at its progress via the sandbox's preview tool and adjust its code in the next turn.
  • State over Pixels: While visual metrics (like CLIP distance) are useful, they can be noisy. BlenderFORGE uses "State Verifiability"—directly checking the scene graph for object positions or material values—to provide much more stable reward signals for RL.
  • Amortizing Search: By training on these trajectories, the "experience" of searching for the right edit is baked into the model's weights. This makes the final agent much faster and more efficient at deployment than systems that have to search from scratch every time.
Figure 2: The BlenderFORGE architecture. BlenderFORGE is a unified framework that integrates a scalable 3D asset-based data generation pipeline with an interactive Blender sandbox to enable training MLLM agents in 3D graphics editing. (Left) Data Generation Pipeline: 3D assets are perturbed and filtered through a visibility check to ensure that the changes are observable. (Right) Training Loop: The validated data flows into training with an interactive Blender sandbox, enabling SFT and RL for MLLM agents.

Key Results

The researchers trained two primary agents: Qwen3-VL-8B-BlenderFORGE and Qwen3.5-9B-BlenderFORGE.

  • Massive Gains over Base Models: On the object placement task, the success rate ([email protected]) for the 9B model jumped from a measly 3.00% (base) to 40.00% after training.
  • Competitive with Proprietary Models: The trained open-weight agents outperformed several zero-shot proprietary models and ranked just behind GPT-5.2 and Claude-Sonnet-4.6 on placement and lighting metrics.
  • Generalization: The models showed strong transfer performance on the independent BlenderGym benchmark and even on "held-out" tasks like camera reframing, which were not part of the original training distribution.

Limitations and Future Work

  • Task Complexity: The current framework focuses on four specific task families. It does not yet handle more complex operations like object insertion/deletion, physics-aware editing, or long-horizon scene redesign.
  • Camera Control: While the agents improved, general-purpose camera navigation remains a challenge, with a "valid rate" of only 35% in held-out tests.
  • Environment Specificity: The agents are currently "Blender-native." Transferring these skills to other engines (like Unreal or Unity) or CAD systems remains an open research question.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have shown promise as interactive agents, yet precise, programmatic 3D graphics editing remains difficult to train because existing systems largely rely on closed-source model APIs and lack deterministic, state-verifiable execution environments. We present BlenderFORGE, a trainable data-environment-optimization framework for open-weight MLLM agents that edit explicit Blender scenes through Blender Python (bpy) code generation. BlenderFORGE targets four fundamental state-verifiable editing task families: object placement, shape-key editing, lighting adjustment, and material modification. It integrates three components: (1) a scalable perturbation-based data generation pipeline built from curated assets and large-scale 3D scene repositories; (2) an isolated Blender Sandbox for deterministic code execution, structured state export, and closed-loop reward computation; and (3) a two-stage training pipeline that uses offline teacher trajectories for cold-start Supervised Fine-Tuning (SFT) and sandbox-computed rewards for multi-turn tool-agentic Reinforcement Learning (AgentLoop-RL). Experiments across the four task families show that the resulting open-weight Qwen3-VL-8B and Qwen3.5-9B BlenderFORGE agents substantially improve over their base models and achieve competitive performance with several zero-shot proprietary MLLM baselines on selected metrics. Additional cross-benchmark evaluations and camera-reframing experiments provide evidence of transfer beyond the original training setting.
BibTeX
@misc{guo2026blenderforge,
  author = {Zilin Guo and Tianrui Zhang and Yi Rong and Yichen Liu and Yuxin Guo and Jingcheng Ni and Lewei Lu and Zehuan Wu and Dan Xu},
  title = {{BlenderFORGE: Framework for Optimizing Reactive 3D-Graphics Editing Ability of MLLMs}},
  year = {2026},
  eprint = {2611.10768},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10768},
}
Figure