TL;DR
BWM-Forge is a code-driven framework that generates highly controllable robot videos by translating task descriptions into editable 3D scene programs. By using these programs to create "proxy videos" that guide high-fidelity generative models, it allows researchers to precisely manipulate robot motion, viewpoints, and scene geometry—bridging the gap between rigid simulations and "black-box" video generation.
What Problem Does This Paper Address?
Training embodied AI (robots) requires massive amounts of diverse video data. While end-to-end video generation models can produce realistic footage, they are notoriously difficult to control. If a generated robot misses a grasp by an inch, you can't easily "nudge" the motion in the latent space; you usually have to regenerate the entire video, often losing scene consistency.
Conversely, traditional robotics simulators provide perfect control but often suffer from a "sim-to-real" gap where the visuals are too simplistic for training robust vision-based policies. BWM-Forge tackles this by decoupling structure (geometry and motion) from appearance (pixels and textures), providing a "best of both worlds" approach for generating synthetic training data.
Methods
The core of BWM-Forge is an Agentic Harness—a multi-agent system that coordinates the transition from a text prompt or human demonstration to a final high-fidelity video.
- Preprocessing Agent: Validates robot assets (URDF files) and parses input data (images or demo videos) into a semantic scene state.
- Building Agent: This is the "programmer." It writes Blender scripts to construct a 3D scene, import the robot, and animate its motion. This results in an editable scene program.
- Review Agent: Acts as a critic, inspecting the 3D proxy for collisions, "floating" objects, or motion glitches, and applying local code corrections.
- Rendering Agent: Takes the "Proxy Video" (a simplified 3D render showing geometry and motion) and uses a high-fidelity video model (Seedance) to synthesize realistic textures, lighting, and shadows.
A standout feature is the Experience Repository, which stores successful modeling and rendering rules. If the system learns how to handle "mesh interpenetration" during a grasping task in a workshop, it can retrieve and apply that logic to a similar task in a kitchen later on.
Datasets
The authors evaluated BWM-Forge on 15 diverse tasks covering five distinct robot embodiments:
- Single-arm manipulators
- Dual-arm systems with grippers
- Dual-arm systems with dexterous hands
- Quadrupeds (four-legged robots)
- Humanoids
The scenes span everyday environments like kitchens and offices to industrial warehouses and surgical training setups. They compared their results against LingBot-World Base Cam, a state-of-the-art baseline for robot video generation.
Key Insights and Ideas
- Code as the Intermediate Representation: By representing the world as executable Blender code rather than just pixels, the system makes the "world state" explicit and addressable. You can change the camera angle or swap a robot arm for a humanoid just by modifying a few lines of code while keeping the rest of the scene intact.
- Synchronized Multi-View Generation: Because the underlying scene program is a 3D model, BWM-Forge can render multiple "proxy videos" from different camera angles for the exact same motion timeline. This ensures perfect temporal and spatial alignment across views, which is critical for training multi-camera robot policies.
- Feedback-Driven Refinement: The framework supports an iterative loop. A user (or the Review Agent) can provide feedback like "move the cup to the right," and the Building Agent updates the code specifically for that entity without destroying the rest of the generated content.
Key Results
BWM-Forge outperformed the LingBot-World baseline across nearly all VBench-Long metrics:
- Subject Consistency: 0.972 vs. 0.948 (indicating the robot's appearance stays stable).
- Motion Smoothness: 0.994 vs. 0.987.
- Physical Adherence (RoboJudge-9B): 2.93 vs. 2.89.
- Instruction Alignment (RoboJudge-9B): 3.33 vs. 2.73 (a significant 0.6-point lead).
Qualitatively, the system demonstrated a remarkable ability to maintain task structure even when drastically changing the environment's appearance (e.g., turning a warehouse into a vertical farm) or the robot's embodiment.
Limitations and Future Work
While powerful, BWM-Forge currently has a few hurdles:
- Human-in-the-loop: Some complex refinements still benefit from manual user feedback, though the goal is full automation.
- Scale: The evaluation was limited to 15 tasks. Scaling this to thousands of automated tasks is the next logical step.
- Downstream Validation: The paper focuses on generating the videos; the authors note that future work is needed to establish exactly how much these generated videos improve the performance of downstream robot learning policies.
Summary updated