TL;DR
RoboCoEvo is a framework that creates a virtuous cycle between a robotic policy and an MLLM-based agent. By having the agent identify recurring failure modes and write executable recovery code, the system generates targeted corrective data that the policy "internalizes" through fine-tuning, leading to sustained performance gains across successive rounds of co-evolution.
What Problem Does This Paper Address?
Current robotic systems often combine a low-level policy (like a Vision-Language-Action model) with a high-level agent (like an MLLM) that acts as a supervisor. However, these two components are usually developed in isolation.
This decoupling leads to two main issues:
- Dependency: The policy never learns from the agent’s interventions, remaining dependent on the agent to fix the same mistakes repeatedly.
- Stagnation: The agent’s knowledge remains static. As the policy improves and its failure patterns shift, a static agent continues to target weaknesses that may no longer exist while missing new, emerging bottlenecks.
RoboCoEvo addresses this by treating policy learning and agent adaptation as a coupled, iterative process.
Methods
The RoboCoEvo framework operates in two main phases that repeat in "rounds":
- Policy-Guided Agent Evolution:
- The system collects rollouts from the current policy and identifies failures.
- An MLLM agent groups these failures into Failure Modes (FMs) based on recurring patterns (e.g., "slipping while grasping a jar").
- The agent then writes Detectors (to find these states in future rollouts) and Recovery Programs (code-as-policy scripts that use motion planners to complete the failed subtask).
- Agent-Guided Policy Capability Expansion:
- Failure-Conditioned Augmentation: The system takes a failed state and perturbs it in the simulator (changing positions slightly) while ensuring the failure mode remains the same.
- The recovery programs are run on these states to generate a large volume of "corrective demonstrations."
- The policy is fine-tuned on a mix of original successful demonstrations and this new corrective data, allowing it to "internalize" the recovery behavior.
In the next round, the updated policy is deployed, and the agent revises its programs to address the new distribution of failures.
Datasets
The authors evaluated RoboCoEvo on three diverse simulation benchmarks:
- BEHAVIOR-1K: Challenging, long-horizon bimanual mobile manipulation tasks in household settings.
- RoboTwin: Bimanual tabletop manipulation tasks requiring high coordination.
- RoboCasa: Realistic kitchen environments using a single-arm mobile manipulator.
Key Insights and Ideas
- Targeted Data is King: The researchers found that training on data specifically targeting the policy's failures is significantly more effective than simply adding more successful demonstrations. For example, "success-only" data required nearly double the GPU hours and significant human effort to achieve lower performance than the automated RoboCoEvo approach.
- Consistency via Shared Programs: Instead of generating a unique recovery for every single failure, grouping failures into "modes" and using a shared recovery program produces more consistent data. This consistency makes it much easier for the neural network policy to learn the underlying pattern.
- The "Moving Bottleneck" Phenomenon: In long-horizon tasks (like "canning meat"), the policy's progress is often hidden. In Round 1, the policy might fail to even get a jar out of a cabinet. Once it learns that, Round 2 reveals new failures in opening the lid. Co-evolution is necessary to chase these shifting bottlenecks down the task chain.
- Efficiency through Augmentation: By perturbing failed states in the simulator, the system can generate 3.6x more training data without needing to run more expensive, full-task rollouts.
Key Results
RoboCoEvo demonstrated sustained improvements across all benchmarks over two rounds of evolution:
- RoboTwin: Success rates jumped from 79% to 93%.
- RoboCasa: Success rates increased from 79% to 91%.
- BEHAVIOR-1K: The mean Q-score (a measure of subtask completion) rose from 0.40 to 0.56.
- Failure Reduction: The fraction of episodes failing due to "Targeted Failure Modes" dropped drastically (e.g., from 0.188 to 0.058 on RoboTwin), proving the policy successfully internalized the agent's corrections.
Limitations and Future Work
- Success Bias: Currently, the framework only learns from successful agent recoveries. It ignores the potential information contained in failed recovery attempts.
- Lack of Reinforcement Learning: The system relies on imitation learning (DAgger-style). Integrating Reinforcement Learning could allow the policy to explore and optimize beyond the specific recovery programs written by the agent.
- Simulation Dependency: The current failure-conditioned augmentation relies on having access to the simulator state to perturb objects and robots, which may be harder to implement in real-world settings without high-fidelity digital twins.
Summary updated