TL;DR
This paper introduces CTS, a novel concurrent reinforcement learning framework that trains teacher and student policies simultaneously for legged robot locomotion. By jointly optimizing both policies, CTS achieves more robust and agile locomotion on challenging terrains compared to traditional sequential teacher-student approaches, reducing velocity tracking errors by up to 20%.
What Problem Does This Paper Address?
Legged locomotion for robots is a notoriously difficult problem due to complex interactions with uneven terrain and inherent nonlinear dynamics. While reinforcement learning (RL) has shown great promise, achieving robust and agile locomotion, especially in real-world scenarios with limited sensory information (proprioception only), remains a challenge. Traditional "teacher-student" methods train a "teacher" policy with access to privileged information (like terrain maps) and then train a "student" policy to mimic it using only proprioceptive data. This sequential approach can lead to suboptimal performance for the student, as it's trying to learn from a potentially imperfect teacher or a moving target. This paper aims to bridge this gap by developing a more efficient and effective concurrent training method.
Methods
The core innovation is the Concurrent Teacher-Student (CTS) architecture. Unlike traditional methods, CTS trains the teacher and student policies concurrently within a single reinforcement learning framework, specifically using a modified Proximal Policy Optimization (PPO) algorithm.
Here's a breakdown of the key components:
- Concurrent Training: Both teacher and student policies are trained simultaneously. The teacher policy has access to privileged information (full state), while the student policy relies solely on proprioceptive observations.
- Modified PPO: The training scheme leverages data samples collected from interactions of both the teacher and student policies with the environment. This allows for a more efficient use of experience.
- Asymmetric Actor-Critic: The framework uses an asymmetric actor-critic setup. The teacher policy includes a privileged encoder and a policy network, while the student policy has a proprioceptive encoder and a policy network. They share a common critic network.
- Proprioceptive Encoder Training: The student's proprioceptive encoder is trained not only through imitation but also by minimizing a reconstruction loss against the teacher's privileged encoder output. This helps the student learn a more meaningful latent representation from limited sensory input.
- Reward Design: A comprehensive reward function is designed, tailored for both quadrupedal and bipedal robots, including terms for velocity tracking, stability, and specific gait characteristics. For the bipedal robot, additional rewards are introduced to address its underactuation and balance challenges.
Datasets
The paper does not explicitly mention specific pre-existing datasets for training or evaluation. Instead, it relies on:
- Simulated Environments: The training is conducted in the IsaacGym simulator, utilizing a curriculum of increasingly difficult terrains (slopes, rough slopes, stairs, discrete obstacles) with randomized parameters to bridge the sim-to-real gap.
- Real-World Experiments: The trained student policies are deployed on physical quadruped robots (Unitree A1, Aliengo) and a point-foot bipedal robot (LimX Dynamics P1) for validation.
Key Insights and Ideas
- Concurrent Training is Key: The paper strongly suggests that training teacher and student policies simultaneously, rather than sequentially, leads to better performance for the student. This allows the student to learn from a more stable and evolving teacher, and vice-versa, creating a synergistic learning process.
- Beyond Imitation: Guiding the student's training with RL objectives, in addition to supervised learning for latent representation reconstruction, is crucial. This prevents the student from being solely reliant on potentially imperfect imitation and encourages it to learn a truly optimal policy for its limited sensory input.
- Bridging the Information Gap: The concurrent approach effectively leverages the teacher's privileged information to guide the student's learning, even though the student never directly uses this information during deployment. This is a more efficient way to transfer knowledge than traditional supervised imitation.
- Universality of the Framework: The CTS framework demonstrates its effectiveness across different robot morphologies (quadrupeds of various sizes and a bipedal robot) and challenging terrains, highlighting its generalizability.
Key Results
- Reduced Velocity Tracking Error: CTS achieved up to a 20% reduction in average velocity tracking error on uneven terrains compared to the two-stage teacher-student method. Specific improvements ranged from 7.9% to 21.85% across different terrain types.
- Enhanced Robustness: CTS demonstrated significantly greater robustness against random pushes, with survival rates higher than the two-stage teacher-student method by 4.57% to 6.47%.
- Comparable Teacher Performance: The teacher policy in CTS achieved performance nearly identical to an "Oracle" policy (trained with full state information), indicating that concurrent training does not hinder the teacher's learning.
- Real-World Success: Extensive real-world experiments on physical robots showcased robust and agile locomotion capabilities, including navigating moving platforms, stairs, and recovering from stumbles.
Limitations and Future Work
- Physical Leg-Obstacle Interaction: The authors acknowledge that the current approach requires physical interaction with obstacles for adaptation.
- Future Work - Exteroceptive Inputs: Future efforts will focus on incorporating exteroceptive inputs (e.g., vision) into the locomotion system. This aims to refine gait planning and obstacle negotiation strategies before physical contact occurs, further enhancing proactive navigation.
Summary updated