TL;DR
CoachSI is a self-improvement framework that enables robot policies to master complex, multi-step tasks by using a frozen Vision-Language Model (VLM) as a "coach." By simply providing the correct next-step instruction when a robot stalls, the coach generates successful trajectories that, when used for fine-tuning, significantly boost unassisted performance beyond what is possible with human demonstrations alone.
What Problem Does This Paper Address?
Modern generalist robot policies are surprisingly adept at "atomic" skills—simple actions like picking up an object or opening a drawer. However, they frequently stumble when tasked with long-horizon challenges that require chaining these skills together (e.g., "get the mug, put it in the microwave, and close the door").
The authors identify a specific failure mode: the stall. Often, a robot stops making progress not because it lacks the physical capability to continue, but because it fails to invoke the correct next skill. Traditional self-improvement—where a robot trains on its own successful attempts—fails here because the robot rarely succeeds at these complex tasks on its own, leaving it with no "good" data to learn from.
Methods
CoachSI introduces a three-stage pipeline for autonomous improvement:
- Coached Collection: A robot attempts a long-horizon task. If it stalls, a frozen, off-the-shelf VLM (the "coach") analyzes the camera feed and selects the appropriate "next-step" instruction from a candidate set of atomic skills. This "hint" often allows the robot to overcome the stall and complete the task.
- Hindsight Relabeling: Successful coached rollouts are saved. Crucially, the specific "next-step" instructions provided by the coach are stripped away and replaced with the original high-level task instruction. This teaches the policy that the high-level goal implies this successful sequence of actions.
- Backbone-Specific Fine-tuning: The policy is fine-tuned on this relabeled data. For World Action Models (WAMs), the authors only update the video backbone (responsible for predicting "what happens next" visually) while keeping the action backbone (responsible for motor control) frozen.
Datasets
The researchers evaluated CoachSI using RoboCasa365, a large-scale simulation framework built on top of RoboCasa. It provides diverse, realistic kitchen environments. The evaluation focused on 18 long-horizon tasks tested across 20 unseen scenes to ensure the learned improvements generalize to new environments.
Key Insights and Ideas
- Stalls are Instructional, Not Physical: The core "aha!" moment is that robots often have the skills they need but don't know when to use them. Providing the correct next-step text resolved stalls 4.6x more effectively than repeating the high-level task instruction.
- Language is a Low-Bandwidth, High-Impact Signal: Unlike interactive imitation learning which requires a human to provide precise motor corrections (teleoperation), CoachSI shows that a simple language "nudge" from a VLM is a sufficient supervisory signal for self-improvement.
- Improving the "Mind's Eye" is Enough: By freezing the action backbone during training, the authors proved that the robot didn't need to learn new ways to move. It simply needed to improve its internal prediction of what the scene should look like next to trigger the skills it already possessed.
- Distillation Outperforms Interaction: Surprisingly, the fine-tuned policy performed better on its own than the original policy did with the coach actively helping it at test time. This suggests that fine-tuning effectively "bakes in" the coach's logic, making the policy more robust.
Key Results
- Resolving Stalls: On states where the robot had stalled, the VLM coach's instructions increased subgoal completion from 2.9% to 13.5%.
- Boosting Success Rates: CoachSI improved unassisted success on long-horizon tasks from 47.4% to 59.3%, an 11.9 point gain.
- Outperforming Baselines: At an equal data budget, CoachSI's self-improvement outperformed fine-tuning on human demonstrations by 9.8 points and fine-tuning on the robot's own uncoached successes by 7.3 points.
- VLM Efficiency: A VLM coach using only camera images was highly effective, recovering 91% of the performance gain achieved by an "oracle" coach with perfect state information.
Limitations and Future Work
- Candidate Skill Sets: The current implementation requires a predefined list of atomic instructions for the VLM to choose from. Future iterations could allow the VLM to generate open-ended instructions.
- Sim-to-Real: While the results in simulation are compelling, the transition to physical hardware remains a critical next step.
- Heuristic Stall Detection: The method currently uses a fixed time-out to detect stalls. Developing a learned model to identify when a robot is "stuck" could make the data collection process much faster and more autonomous.
Summary updated