Abstract
Pre-trained Vision-Language-Action (VLA) models represent a major leap towards general-purpose robots, yet efficiently adapting them to novel, specific tasks in-situ remains a significant hurdle. While reinforcement learning (RL) is a promising avenue for such adaptation, the process often suffers from low efficiency, hindering rapid task mastery. We introduce Reflective Self-Adaptation, a framework for rapid, autonomous task adaptation without human intervention. Our framework establishes a self-improving loop where the agent learns from its own experience to enhance both strategy and execution. The core of our framework is a dual-pathway architecture that addresses the full adaptation lifecycle. First, a Failure-Driven Reflective RL pathway enables rapid learning by using the VLM's causal reasoning to automatically synthesize a targeted, dense reward function from failure analysis. This provides a focused learning signal that significantly accelerates policy exploration. However, optimizing such proxy rewards introduces a potential risk of "reward hacking," where the agent masters the reward function but fails the actual task. To counteract this, our second pathway, Success-Driven Quality-Guided SFT, grounds the policy in holistic success. It identifies and selectively imitates high-quality successful trajectories, ensuring the agent remains aligned with the ultimate task goal. This pathway is strengthened by a conditional curriculum mechanism to aid initial exploration. We conduct experiments in challenging manipulation tasks. The results demonstrate that our framework achieves faster convergence and higher final success rates compared to representative baselines. Our work presents a robust solution for creating self-improving agents that can efficiently and reliably adapt to new environments.
Introduction
While large-scale foundation models have provided robots with impressive "zero-shot" capabilities, they often struggle with "last-mile" adaptation in specific, real-world settings. A robot might understand the general concept of "picking up a bowl," but fail in a specific environment due to local constraints like clutter or surface properties. Traditional Reinforcement Learning (RL) is often used for such adaptation but faces two primary challenges: sparse rewards, which make learning slow, and reward hacking, where agents exploit proxy rewards without completing the actual task.
Methodology: The Reflective Self-Adaptation Framework
The proposed framework addresses these challenges through a dual-pathway architecture that creates a self-improving closed loop.
1. Failure-Driven Reflective RL
When the agent fails, it utilizes a Vision-Language Model (VLM) to perform Reflective Reward Synthesis. This process involves:
- Causal Analysis: The VLM analyzes failure videos to identify the primary cause of failure (e.g., "collided with the basket").
- Component Selection: It selects corrective behaviors from a predefined library (e.g.,
avoid(obstacle)). - Relationship Identification: It defines the logical relationships between these components (e.g., AND, IF, OR).
- Configuration: It generates a structured JSON configuration for a new, dense reward function used for Proximal Policy Optimization (PPO).
2. Success-Driven Quality-Guided SFT
To mitigate reward hacking, the framework identifies successful attempts for Supervised Fine-Tuning (SFT).
- Quality Assessment: Trajectories are scored based on efficiency and alignment with reflective rewards, prioritizing the highest-quality data for imitation.
- Conditional Curriculum: For tasks with a high "cold start" barrier, the VLM acts as an automated programmer, simplifying simulation environments (e.g., removing obstacles) to facilitate initial exploration.
Experimental Evaluation
The framework was evaluated on the LIBERO Benchmark (Spatial, Object, Goal, and Long-horizon tasks) and a new LIBERO-Adapt suite. LIBERO-Adapt consists of 10 manipulation scenarios specifically designed to be difficult for pre-trained models, providing a rigorous test for in-situ adaptation.
Results and Discussion
The framework achieved an average success rate of 83.6% across LIBERO tasks, outperforming baselines like VLA-RL (81.0%) and Octo (75.1%). In the LIBERO-Adapt suite, it demonstrated significantly faster convergence and higher stability compared to standard RL methods, which often suffered from performance collapses due to reward hacking.
Key insights from the study include:
- VLM as an In-the-Loop Reasoner: Moving beyond high-level planning, the VLM serves as an automated reward engineer, turning reward design into a structured reasoning task.
- Synergy of Pathways: Failures provide causal reasoning ("why"), while successes provide behavioral templates ("how"). Ablation studies confirmed that both pathways are essential for optimal performance.
Conclusion
Reflective Self-Adaptation offers a robust solution for creating self-improving robotic agents. While currently dependent on high-end VLMs and a predefined reward library, future work will explore the use of smaller open-source models and the autonomous generation of entirely new reward code. This approach marks a significant step toward robots that can efficiently and reliably adapt to any new environment.
Summary updated