CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

Yijia Fan, Xilin He, Yilei Jiang, Dongming Wu, Jiaxu Wang, Xiangyu YuearXiv 2026 · arXiv

TL;DR

CaPtain is a post-training pipeline that transforms small, open-source vision-language models (VLMs) into capable "Code-as-Policy" robot agents. By combining supervised fine-tuning with a reinforcement learning stage guided by a "privileged" simulator critic, a 4B-parameter model can outperform massive frontier models like GPT-4 at a fraction of the cost, while demonstrating remarkable zero-shot transfer to real-world hardware.

What Problem Does This Paper Address?

Most current "Code-as-Policy" (CaP) agents—robots that perform tasks by writing and executing Python code—rely on expensive, closed-source frontier models (like GPT-4) or delegate the actual "heavy lifting" to pre-trained Vision-Language-Action (VLA) models. Small, open-source VLMs typically fail at this task because they struggle to map high-level instructions to low-level robot APIs.

Furthermore, training these agents using Reinforcement Learning (RL) is notoriously difficult. A robot might make dozens of code calls in a single episode, but it only receives a "success" or "failure" signal at the very end. Figuring out which specific line of code led to success (the credit assignment problem) is a major bottleneck for multi-turn agents.

Methods

The authors introduce CaPtain, a two-stage post-training pipeline:

  1. Stage 1: Supervised Fine-Tuning (SFT): The base model (Qwen3.5-4B) is trained on synthetic trajectories generated by a "teacher" model (GPT-6 Astra). This teaches the model the basic syntax of the CaPtain-Harness, a sandboxed environment with 22 low-level primitives for perception, geometry, and motion.
  2. Stage 2: RL with a Physical Critic: To solve the credit assignment problem, the authors use an asymmetric actor-critic approach. While the "actor" (the robot policy) only sees camera images, a "physical critic" is trained in simulation with access to the privileged state (exact object coordinates, grasp status, etc.).
Overview of the CaPtain pipeline, showing the inference loop in the harness and the two-stage training process.

The critic predicts the probability of success at every step. Credit is assigned to a request based on how much it changes this predicted probability. This ensures that physical actions like "grasping" get the credit they deserve, while simple "perception" calls receive near-zero credit.

Datasets

  • Training: 40 tasks from the LIBERO benchmark (spatial, object, goal, and long-horizon manipulation).
  • Evaluation:
    • LIBERO-PRO, LIBERO-Plus, and LIBERO-MAX: Three perturbation suites that test robustness against changes in object positions, lighting, and camera angles.
  • Zero-shot Transfer: Evaluated on RoboTwin 2.0, RoboDojo, SimplerEnv (WidowX), and CaP-Bench.
  • Real-world: Tested on an Astrabot XR1 arm using only simulation-based adaptation.

Key Insights and Ideas

  • Low-Level is Better: Unlike many agents that call high-level "skills" (like grasp()), CaPtain writes code for raw end-effector poses and gripper commands. This gives the RL process full control over the manipulation strategy.
  • The Power of Asymmetry: Using a critic that "cheats" by looking at the simulator's internal state allows a small model to learn complex physical nuances that it couldn't infer from images alone.
  • Code vs. End-to-End: The authors find that code-writing agents are significantly more robust to visual perturbations than end-to-end VLA models. Because the code relies on geometric measurements and feedback "receipts," it can adapt to a camera being moved or lighting changing more easily than a model that maps pixels directly to motor torques.
  • Concentrated Credit: Analysis showed that 93% of the RL credit mass was correctly assigned to physical motions and grasps, while perception turns accounted for only 1.3%, validating the effectiveness of the physical critic.

Key Results

  • Performance Leap: Starting from 0% success, CaPtain-4B reached 73.8% success on held-out LIBERO tasks.
  • Beating the Giants: On the LIBERO-PRO benchmark, CaPtain-4B achieved 78.5% success, surpassing several frontier-model agents (like those using Gemini or Claude) and much larger open models.
  • Robustness: While standard end-to-end models often see performance tank under perturbations (e.g., dropping from 97% to 12%), CaPtain's RL gains persisted across all tested scenarios.
  • Real-World Transfer: Without any real-world training data, the model successfully performed tasks on a physical robot arm by adapting through a "kinematic twin" simulation.
Comparison of success rates across different LIBERO benchmarks, showing the significant gains from RL.

Limitations and Future Work

  • Bimanual Coordination: The current model was trained on single-arm tasks. While it can handle some dual-arm setups via simple heuristics, it lacks true simultaneous bimanual coordination.
  • Privileged State Dependency: The training process requires a high-fidelity simulator to provide the critic with exact object poses. Moving this RL process to real hardware would require an external state estimator.
  • Action Space: The 22 primitives are based on "guarded" Cartesian motion. This prevents the model from performing tasks that require continuous force-controlled compliance, such as wiping a surface or turning a stiff valve.

Summary updated

Abstract

Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply. Yet existing CaP agents sidestep training: they either prompt closed frontier models or delegate manipulation to pre-trained vision-language-action models (VLAs) and high-level skills, while small open vision-language models (VLMs) fail outright. We present CaPtain, an open post-training pipeline that turns a small open VLM into a capable CaP agent, with no VLA, frontier model, or privileged state at test time. CaPtain trains and deploys the policy inside CaPtain-Harness, which exposes low-level perception, geometry, and guarded-motion primitives across simulation and real hardware. We first align the base model via SFT on synthetic trajectories generated with GPT-6 Astra, then apply multi-turn RL. Multi-turn RL must credit tens of agent requests, whose physical effects often surface turns later, from a single end-of-episode outcome. We therefore introduce a physical critic that reads privileged simulator state, never seen by the policy, and credits each request by how much it changes the predicted chance of success. Starting from Qwen3.5-4B (0% zero-shot), CaPtain-4B reaches 55.4% held-out LIBERO success after SFT and 73.8% after RL. On LIBERO-PRO, it reaches 78.5%, surpassing several frontier-model agents, and its RL gains persist across three perturbation suites (+17.6 to +21.2 points). Without target-domain training, it attains 83.3% on SimplerEnv WidowX, exceeds frontier models on CaP-Bench, and runs on a real robot after simulation-only adaptation. We release all code, data, and model weights.
BibTeX
@misc{fan2026captain,
  author = {Yijia Fan and Xilin He and Yilei Jiang and Dongming Wu and Jiaxu Wang and Xiangyu Yue},
  title = {{CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents}},
  year = {2026},
  eprint = {2610.40799},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40799},
}

Loading related content...

Figure