Abstract
Autonomous skill discovery aims to enable robots to acquire diverse behaviors without explicit supervision. Learning such behaviors directly on physical hardware remains challenging due to safety and data efficiency constraints. Existing methods, including Quality-Diversity Actor-Critic (QDAC), require manually defined skill spaces and carefully tuned heuristics, limiting real-world applicability. We propose Unsupervised Real-world Skill Acquisition (URSA), an extension of QDAC that enables robots to autonomously discover and master diverse, high-performing skills directly in the real world. We demonstrate that URSA successfully discovers diverse locomotion skills on a Unitree A1 quadruped in both simulation and the real world. Our approach supports both heuristic-driven skill discovery and fully unsupervised settings. We also show that the learned skill repertoire can be reused for downstream tasks such as real-world damage adaptation, where URSA outperforms all baselines in 5 out of 9 simulated and 3 out of 5 real-world damage scenarios. Our results establish a new framework for real-world robot learning that enables continuous skill discovery with limited human intervention, representing a significant step toward more autonomous and adaptable robotic systems.
Introduction
In traditional robotics, agents are typically trained to optimize a single objective, such as walking forward. While effective in controlled settings, these "optimal" gaits often fail when the robot encounters novel environments or mechanical damage. To build resilient robotic systems, researchers utilize Quality-Diversity (QD) algorithms, which aim to discover a "repertoire" of diverse, high-performing behaviors. However, traditional QD methods face significant hurdles for real-world deployment: they often require manually defined behavior spaces, massive amounts of data, and involve high-risk exploration that can damage physical hardware.
The proposed URSA (Unsupervised Real-world Skill Acquisition) framework addresses these challenges, enabling safe, sample-efficient, and unsupervised skill discovery directly on physical hardware.
Methodology: The URSA Framework
URSA integrates three core components to facilitate real-world learning:
- World Models (DayDreamer): To overcome the data inefficiency of physical trials, URSA employs a world model. This allows the robot to "imagine" the consequences of its actions, practicing thousands of skill variations in a latent simulation before executing them in the real world.
- Unsupervised Skill Discovery: Instead of relying on human-defined heuristics for diversity, URSA utilizes a Variational Autoencoder (VAE). The VAE compresses raw sensor data—such as joint angles and body orientation—into a low-dimensional latent space. The robot then explores this space to discover a wide variety of behaviors.
- Constrained Quality-Diversity (QDAC): URSA optimizes for both performance (quality) and novelty (diversity) within the latent map. A critical addition is the Safety Critic, which acts as a predictive guardrail, preventing the robot from attempting behaviors that the world model forecasts will lead to falls or hardware failure.
Experimental Setup and Evaluation
The framework was evaluated using a Unitree A1 quadruped robot. Data was collected through:
- Simulation: 1 million timesteps in PyBullet.
- Real-World: 5 hours of continuous interaction.
To test adaptability, the authors benchmarked the robot across 9 simulated and 5 real-world "damage" scenarios, such as locking joints or disabling motors.
Results and Key Insights
URSA demonstrated a significant improvement over standard reinforcement learning baselines, achieving 4x greater coverage of the joint-angle space. The robot successfully learned a "menu" of movement styles, ranging from crouching shuffles to agile trots, in just five hours of real-world training.
Resilience and Adaptation
When subjected to mechanical damage, URSA utilized Iterative Trial and Error (ITE) to search its learned repertoire for a viable gait. It outperformed all baselines in the majority of damage scenarios (5/9 simulated, 3/5 real-world).
Safety as an Enabler
A key insight from this work is that safety constraints do not necessarily hinder exploration; rather, they enable it. By providing a safety critic, URSA allows the robot to aggressively explore diverse behaviors without the risk of catastrophic failure, which is essential for real-world learning.
Discussion and Limitations
While URSA represents a major step forward, several limitations remain:
- Limb Dependency: Because skills are learned via a single neural network, damage to one leg can sometimes disable large portions of the repertoire simultaneously.
- Autonomy Gaps: The system still requires human intervention for "resets" if the robot wanders out of its designated testing area.
- Reward Shaping: The current reward function can lead to "crab-walking" behaviors as it does not strictly penalize lateral drift.
Future work will focus on learning safety boundaries directly from interaction data and improving the autonomy of the reset process to enable truly continuous, long-term skill discovery.
Summary updated