BWM-Forge: Agentic World Programming for Embodied Intelligence

Bwm TeamarXiv 2026 · arXiv

TL;DR

BWM-Forge is a code-driven framework that generates highly controllable robot videos by translating task descriptions into editable 3D scene programs. By using these programs to create "proxy videos" that guide high-fidelity generative models, it allows researchers to precisely manipulate robot motion, viewpoints, and scene geometry—bridging the gap between rigid simulations and "black-box" video generation.

What Problem Does This Paper Address?

Training embodied AI (robots) requires massive amounts of diverse video data. While end-to-end video generation models can produce realistic footage, they are notoriously difficult to control. If a generated robot misses a grasp by an inch, you can't easily "nudge" the motion in the latent space; you usually have to regenerate the entire video, often losing scene consistency.

Conversely, traditional robotics simulators provide perfect control but often suffer from a "sim-to-real" gap where the visuals are too simplistic for training robust vision-based policies. BWM-Forge tackles this by decoupling structure (geometry and motion) from appearance (pixels and textures), providing a "best of both worlds" approach for generating synthetic training data.

Methods

The core of BWM-Forge is an Agentic Harness—a multi-agent system that coordinates the transition from a text prompt or human demonstration to a final high-fidelity video.

  1. Preprocessing Agent: Validates robot assets (URDF files) and parses input data (images or demo videos) into a semantic scene state.
  2. Building Agent: This is the "programmer." It writes Blender scripts to construct a 3D scene, import the robot, and animate its motion. This results in an editable scene program.
  3. Review Agent: Acts as a critic, inspecting the 3D proxy for collisions, "floating" objects, or motion glitches, and applying local code corrections.
  4. Rendering Agent: Takes the "Proxy Video" (a simplified 3D render showing geometry and motion) and uses a high-fidelity video model (Seedance) to synthesize realistic textures, lighting, and shadows.
The BWM-Forge pipeline: from task inputs to high-fidelity video via editable scene programs.

A standout feature is the Experience Repository, which stores successful modeling and rendering rules. If the system learns how to handle "mesh interpenetration" during a grasping task in a workshop, it can retrieve and apply that logic to a similar task in a kitchen later on.

Datasets

The authors evaluated BWM-Forge on 15 diverse tasks covering five distinct robot embodiments:

  • Single-arm manipulators
  • Dual-arm systems with grippers
  • Dual-arm systems with dexterous hands
  • Quadrupeds (four-legged robots)
  • Humanoids

The scenes span everyday environments like kitchens and offices to industrial warehouses and surgical training setups. They compared their results against LingBot-World Base Cam, a state-of-the-art baseline for robot video generation.

Key Insights and Ideas

  • Code as the Intermediate Representation: By representing the world as executable Blender code rather than just pixels, the system makes the "world state" explicit and addressable. You can change the camera angle or swap a robot arm for a humanoid just by modifying a few lines of code while keeping the rest of the scene intact.
  • Synchronized Multi-View Generation: Because the underlying scene program is a 3D model, BWM-Forge can render multiple "proxy videos" from different camera angles for the exact same motion timeline. This ensures perfect temporal and spatial alignment across views, which is critical for training multi-camera robot policies.
  • Feedback-Driven Refinement: The framework supports an iterative loop. A user (or the Review Agent) can provide feedback like "move the cup to the right," and the Building Agent updates the code specifically for that entity without destroying the rest of the generated content.
The iterative building and review workflow allows for targeted local edits.

Key Results

BWM-Forge outperformed the LingBot-World baseline across nearly all VBench-Long metrics:

  • Subject Consistency: 0.972 vs. 0.948 (indicating the robot's appearance stays stable).
  • Motion Smoothness: 0.994 vs. 0.987.
  • Physical Adherence (RoboJudge-9B): 2.93 vs. 2.89.
  • Instruction Alignment (RoboJudge-9B): 3.33 vs. 2.73 (a significant 0.6-point lead).

Qualitatively, the system demonstrated a remarkable ability to maintain task structure even when drastically changing the environment's appearance (e.g., turning a warehouse into a vertical farm) or the robot's embodiment.

Qualitative results showing the proxy model vs. the final rendered frames for a kitchen task.

Limitations and Future Work

While powerful, BWM-Forge currently has a few hurdles:

  • Human-in-the-loop: Some complex refinements still benefit from manual user feedback, though the goal is full automation.
  • Scale: The evaluation was limited to 15 tasks. Scaling this to thousands of automated tasks is the next logical step.
  • Downstream Validation: The paper focuses on generating the videos; the authors note that future work is needed to establish exactly how much these generated videos improve the performance of downstream robot learning policies.

Summary updated

Abstract

Generating diverse robot videos requires control over scene geometry, robot motion, and viewpoints. End-to-end video generation offers limited direct access to these variables, making targeted edits and reuse across tasks difficult. We present BWM-Forge, a code-driven framework for controllable video generation in robotics. Given robot assets and a task description, the framework accepts an initial image, an image with motion data, or a human demonstration video. A coding agent constructs editable Blender scene programs that specify objects, robot motion, and cameras. The resulting proxy videos provide structural and motion references for a video generation model, which adds visual detail. A multi-agent harness coordinates input processing, scene construction, review, and rendering. Automated evaluation and user feedback support iterative refinement, while an experience repository stores and retrieves reusable modeling and rendering rules. This design supports multiple robot embodiments, synchronized viewpoints, and appearance variation from a shared scene program. We evaluate the framework on 15 tasks covering five robot types. Compared with LingBot-World Base Cam on these tasks, BWM-Forge achieves higher aesthetic quality, imaging quality, subject and background consistency, motion smoothness, and RoboJudge-9B physical adherence and instruction alignment scores. The baseline scores higher in dynamic degree. Qualitative cases illustrate viewpoint control, scene variation, feedback-driven edits, and experience reuse across scenes. These results demonstrate a practical approach to generating controllable robot videos through editable, executable scene representations.
BibTeX
@misc{team2026bwm,
  author = {BWM Team},
  title = {{BWM-Forge: Agentic World Programming for Embodied Intelligence}},
  year = {2026},
  eprint = {2611.16221},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16221},
}
Figure