TL;DR
CaPtain is a post-training pipeline that transforms small, open-source vision-language models (VLMs) into capable "Code-as-Policy" robot agents. By combining supervised fine-tuning with a reinforcement learning stage guided by a "privileged" simulator critic, a 4B-parameter model can outperform massive frontier models like GPT-4 at a fraction of the cost, while demonstrating remarkable zero-shot transfer to real-world hardware.
What Problem Does This Paper Address?
Most current "Code-as-Policy" (CaP) agents—robots that perform tasks by writing and executing Python code—rely on expensive, closed-source frontier models (like GPT-4) or delegate the actual "heavy lifting" to pre-trained Vision-Language-Action (VLA) models. Small, open-source VLMs typically fail at this task because they struggle to map high-level instructions to low-level robot APIs.
Furthermore, training these agents using Reinforcement Learning (RL) is notoriously difficult. A robot might make dozens of code calls in a single episode, but it only receives a "success" or "failure" signal at the very end. Figuring out which specific line of code led to success (the credit assignment problem) is a major bottleneck for multi-turn agents.
Methods
The authors introduce CaPtain, a two-stage post-training pipeline:
- Stage 1: Supervised Fine-Tuning (SFT): The base model (Qwen3.5-4B) is trained on synthetic trajectories generated by a "teacher" model (GPT-6 Astra). This teaches the model the basic syntax of the CaPtain-Harness, a sandboxed environment with 22 low-level primitives for perception, geometry, and motion.
- Stage 2: RL with a Physical Critic: To solve the credit assignment problem, the authors use an asymmetric actor-critic approach. While the "actor" (the robot policy) only sees camera images, a "physical critic" is trained in simulation with access to the privileged state (exact object coordinates, grasp status, etc.).
The critic predicts the probability of success at every step. Credit is assigned to a request based on how much it changes this predicted probability. This ensures that physical actions like "grasping" get the credit they deserve, while simple "perception" calls receive near-zero credit.
Datasets
- Training: 40 tasks from the LIBERO benchmark (spatial, object, goal, and long-horizon manipulation).
- Evaluation:
- LIBERO-PRO, LIBERO-Plus, and LIBERO-MAX: Three perturbation suites that test robustness against changes in object positions, lighting, and camera angles.
- Zero-shot Transfer: Evaluated on RoboTwin 2.0, RoboDojo, SimplerEnv (WidowX), and CaP-Bench.
- Real-world: Tested on an Astrabot XR1 arm using only simulation-based adaptation.
Key Insights and Ideas
- Low-Level is Better: Unlike many agents that call high-level "skills" (like
grasp()), CaPtain writes code for raw end-effector poses and gripper commands. This gives the RL process full control over the manipulation strategy. - The Power of Asymmetry: Using a critic that "cheats" by looking at the simulator's internal state allows a small model to learn complex physical nuances that it couldn't infer from images alone.
- Code vs. End-to-End: The authors find that code-writing agents are significantly more robust to visual perturbations than end-to-end VLA models. Because the code relies on geometric measurements and feedback "receipts," it can adapt to a camera being moved or lighting changing more easily than a model that maps pixels directly to motor torques.
- Concentrated Credit: Analysis showed that 93% of the RL credit mass was correctly assigned to physical motions and grasps, while perception turns accounted for only 1.3%, validating the effectiveness of the physical critic.
Key Results
- Performance Leap: Starting from 0% success, CaPtain-4B reached 73.8% success on held-out LIBERO tasks.
- Beating the Giants: On the LIBERO-PRO benchmark, CaPtain-4B achieved 78.5% success, surpassing several frontier-model agents (like those using Gemini or Claude) and much larger open models.
- Robustness: While standard end-to-end models often see performance tank under perturbations (e.g., dropping from 97% to 12%), CaPtain's RL gains persisted across all tested scenarios.
- Real-World Transfer: Without any real-world training data, the model successfully performed tasks on a physical robot arm by adapting through a "kinematic twin" simulation.
Limitations and Future Work
- Bimanual Coordination: The current model was trained on single-arm tasks. While it can handle some dual-arm setups via simple heuristics, it lacks true simultaneous bimanual coordination.
- Privileged State Dependency: The training process requires a high-fidelity simulator to provide the critic with exact object poses. Moving this RL process to real hardware would require an external state estimator.
- Action Space: The 22 primitives are based on "guarded" Cartesian motion. This prevents the model from performing tasks that require continuous force-controlled compliance, such as wiping a surface or turning a stiff valve.
Summary updated