CoachSI: Self-Improvement of Robot Policies with a Language Coach

Jiachen Tao, Gengyu Zhang, Haoxuan Wang, Bing Liu, Yan YanarXiv 2026 · arXiv

TL;DR

CoachSI is a self-improvement framework that enables robot policies to master complex, multi-step tasks by using a frozen Vision-Language Model (VLM) as a "coach." By simply providing the correct next-step instruction when a robot stalls, the coach generates successful trajectories that, when used for fine-tuning, significantly boost unassisted performance beyond what is possible with human demonstrations alone.

What Problem Does This Paper Address?

Modern generalist robot policies are surprisingly adept at "atomic" skills—simple actions like picking up an object or opening a drawer. However, they frequently stumble when tasked with long-horizon challenges that require chaining these skills together (e.g., "get the mug, put it in the microwave, and close the door").

The authors identify a specific failure mode: the stall. Often, a robot stops making progress not because it lacks the physical capability to continue, but because it fails to invoke the correct next skill. Traditional self-improvement—where a robot trains on its own successful attempts—fails here because the robot rarely succeeds at these complex tasks on its own, leaving it with no "good" data to learn from.

Methods

CoachSI introduces a three-stage pipeline for autonomous improvement:

  1. Coached Collection: A robot attempts a long-horizon task. If it stalls, a frozen, off-the-shelf VLM (the "coach") analyzes the camera feed and selects the appropriate "next-step" instruction from a candidate set of atomic skills. This "hint" often allows the robot to overcome the stall and complete the task.
  2. Hindsight Relabeling: Successful coached rollouts are saved. Crucially, the specific "next-step" instructions provided by the coach are stripped away and replaced with the original high-level task instruction. This teaches the policy that the high-level goal implies this successful sequence of actions.
  3. Backbone-Specific Fine-tuning: The policy is fine-tuned on this relabeled data. For World Action Models (WAMs), the authors only update the video backbone (responsible for predicting "what happens next" visually) while keeping the action backbone (responsible for motor control) frozen.
The CoachSI pipeline: (a) Coached collection using a VLM, (b) Hindsight relabeling of successful runs, and (c) Deployment of the improved policy.

Datasets

The researchers evaluated CoachSI using RoboCasa365, a large-scale simulation framework built on top of RoboCasa. It provides diverse, realistic kitchen environments. The evaluation focused on 18 long-horizon tasks tested across 20 unseen scenes to ensure the learned improvements generalize to new environments.

Key Insights and Ideas

  • Stalls are Instructional, Not Physical: The core "aha!" moment is that robots often have the skills they need but don't know when to use them. Providing the correct next-step text resolved stalls 4.6x more effectively than repeating the high-level task instruction.
  • Language is a Low-Bandwidth, High-Impact Signal: Unlike interactive imitation learning which requires a human to provide precise motor corrections (teleoperation), CoachSI shows that a simple language "nudge" from a VLM is a sufficient supervisory signal for self-improvement.
  • Improving the "Mind's Eye" is Enough: By freezing the action backbone during training, the authors proved that the robot didn't need to learn new ways to move. It simply needed to improve its internal prediction of what the scene should look like next to trigger the skills it already possessed.
  • Distillation Outperforms Interaction: Surprisingly, the fine-tuned policy performed better on its own than the original policy did with the coach actively helping it at test time. This suggests that fine-tuning effectively "bakes in" the coach's logic, making the policy more robust.

Key Results

  • Resolving Stalls: On states where the robot had stalled, the VLM coach's instructions increased subgoal completion from 2.9% to 13.5%.
  • Boosting Success Rates: CoachSI improved unassisted success on long-horizon tasks from 47.4% to 59.3%, an 11.9 point gain.
  • Outperforming Baselines: At an equal data budget, CoachSI's self-improvement outperformed fine-tuning on human demonstrations by 9.8 points and fine-tuning on the robot's own uncoached successes by 7.3 points.
  • VLM Efficiency: A VLM coach using only camera images was highly effective, recovering 91% of the performance gain achieved by an "oracle" coach with perfect state information.
Quantitative results showing CoachSI's impact on stalled states and overall task success compared to human demos and self-collection.

Limitations and Future Work

  • Candidate Skill Sets: The current implementation requires a predefined list of atomic instructions for the VLM to choose from. Future iterations could allow the VLM to generate open-ended instructions.
  • Sim-to-Real: While the results in simulation are compelling, the transition to physical hardware remains a critical next step.
  • Heuristic Stall Detection: The method currently uses a fixed time-out to detect stalls. Developing a learned model to identify when a robot is "stuck" could make the data collection process much faster and more autonomous.

Summary updated

Abstract

Generalist robot policies execute atomic skills reliably but fail when the same skills are composed into long-horizon tasks. Self-improvement from the policy's own experience does not close this gap: experience collected by the policy alone stays within its own distribution and reproduces what it can already do. We show that what the policy lacks is not a skill but the next step. We present CoachSI, a self-improvement method in which a frozen, off-the-shelf vision-language model (VLM) coaches the policy in language: it watches the robot and names the next step in the format of the policy's atomic instructions, and never outputs an action. Successful coached rollouts are relabeled in hindsight with the original task instruction and used to fine-tune the policy. The fine-tuned policy is then deployed on its own. On states where a World Action Model and a Vision-Language-Action model have stalled, naming the next step raises subgoal completion by 4.6x over the unchanged task instruction, and the frozen VLM supplies it from the same camera images the policy sees. After fine-tuning, unassisted success on long-horizon tasks in unseen scenes rises from 47.4% to 59.3%, 9.8 points above fine-tuning on human demonstrations and 7.3 points above fine-tuning on self-collected successes at an equal data budget. With a coach that sees only camera images, it reaches 56.8%, above both baselines. The gain is obtained with the action backbone frozen: language coaching teaches the policy what should happen next, and the skills it already has do the rest.
BibTeX
@misc{tao2026coachsi,
  author = {Jiachen Tao and Gengyu Zhang and Haoxuan Wang and Bing Liu and Yan Yan},
  title = {{CoachSI: Self-Improvement of Robot Policies with a Language Coach}},
  year = {2026},
  eprint = {2611.25529},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25529},
}
Figure