Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS Biped

Tianyu Li, Hartmut Geyer, Christopher G. Atkeson, Akshara Rai

TL;DR

This paper tackles the challenge of transferring learned walking controllers from simulation to real-world bipedal robots. They propose a structured approach where a neural network enhances an expert-designed controller, significantly improving transfer rates and maintaining interpretability, unlike purely end-to-end neural network policies.

What Problem Does This Paper Address?

Learning to control complex systems like bipedal robots is notoriously difficult. While deep reinforcement learning (DRL) has shown great promise in simulation, bridging the "sim-to-real" gap remains a major hurdle. This gap arises because simulations are imperfect, and DRL algorithms can exploit these inaccuracies, leading to controllers that fail when deployed on physical hardware. Domain randomization, a common technique to improve transferability, can make the learning process extremely slow and challenging, especially for underactuated robots like the ATRIAS biped, which have a small basin of stability. The authors aim to find a method that allows for effective learning in simulation and successful deployment on hardware without extensive domain randomization or prohibitively long training times.

Methods

The core of this work lies in a "structured neural network policy." Instead of training a monolithic neural network to directly output robot control signals, they integrate a neural network as a component within a pre-existing, expert-designed heuristic controller. This heuristic controller provides a stable framework, defining the overall walking strategy (e.g., maintaining CoM height, torso pitch, and foot placement). The neural network's role is to learn to modulate specific parameters within this heuristic policy, such as desired torso pitch, CoM height, and footstep offsets, based on the robot's state. This approach leverages the strengths of both: the expert controller provides structure and interpretability, while the neural network learns to optimize performance and adapt to challenging conditions. They compare this structured approach against a more traditional, unstructured neural network policy trained end-to-end. For training, they utilize Proximal Policy Optimization (PPO), a stable on-policy DRL algorithm.

Datasets

The experiments were conducted using a high-fidelity simulation of the ATRIAS robot. No specific external datasets were used for training or evaluation; instead, the authors generated their own simulation data through extensive trials.

Key Insights and Ideas

  • Structure is Key for Sim-to-Real: The most significant insight is that introducing structure into learned policies dramatically improves their transferability from simulation to hardware. A purely end-to-end neural network policy, while potentially powerful in simulation, struggles to generalize.
  • Expert Knowledge as a Foundation: Leveraging existing expert knowledge in the form of a heuristic controller provides a robust starting point and a predictable framework for DRL. This prevents the agent from learning unstable or unintuitive behaviors.
  • Interpretability and Modifiability: Structured policies retain a degree of interpretability, allowing users to understand the robot's behavior and even modify parameters without retraining the entire neural network. This is crucial for safety and practical deployment.
  • Faster Learning and Higher Transfer Rates: The structured approach not only leads to better transfer rates but also speeds up the learning process in simulation.

Key Results

  • Transfer Rate Improvement: The structured neural network policy achieved an impressive 80% transfer rate from simulation to hardware, meaning 4 out of 5 trained policies successfully walked on the real ATRIAS robot. In contrast, the unstructured neural network policy only had a 40% transfer rate (2 out of 5).
  • Robustness to Disturbances: The structured policy demonstrated better robustness to ground height disturbances in simulation, a key factor for successful hardware deployment.
  • Faster Simulation Training: The structured policy showed faster reward improvement in simulation compared to the unstructured policy.
  • Hardware Performance: The structured policy successfully enabled the ATRIAS robot to walk on hardware, even overcoming initial challenges like the robot's tendency to jump at the start of a walk.

Limitations and Future Work

  • Initial Conditions Mismatch: A significant challenge identified was the mismatch in initial conditions between simulation and hardware, particularly for starting from rest. This led to controllers failing on hardware, often due to an initial jump.
  • Conservative Controllers: The authors suggest that future work could explore cost functions that encourage more conservative controllers to improve robustness to such initial condition mismatches.
  • Switching Control Frameworks: They propose investigating control frameworks that can dynamically switch between conservative and high-performance controllers based on the perceived mismatch between simulation and hardware.
  • Further Exploration of Structure: While they explored two types of structured policies, there's potential to investigate other structural designs and how they impact learning and transfer.

Summary updated

Abstract

Learning controllers for bipedal robots is a challenging problem, often requiring expert knowledge and extensive tuning of parameters that vary in different situations. Recently, deep reinforcement learning has shown promise at automatically learning controllers for complex systems in simulation. This has been followed by a push towards learning controllers that can be transferred between simulation and hardware, primarily with the use of domain randomization. However, domain randomization can make the problem of finding stable controllers even more challenging, especially for under actuated bipedal robots. In this work, we explore whether policies learned in simulation can be transferred to hardware with the use of high-fidelity simulators and structured controllers. We learn a neural network policy which is a part of a more structured controller. While the neural network is learned in simulation, the rest of the controller stays fixed, and can be tuned by the expert as needed. We show that using this approach can greatly speed up the rate of learning in simulation, as well as enable transfer of policies between simulation and hardware. We present our results on an ATRIAS robot and explore the effect of action spaces and cost functions on the rate of transfer between simulation and hardware. Our results show that structured policies can indeed be learned in simulation and implemented on hardware successfully. This has several advantages, as the structure preserves the intuitive nature of the policy, and the neural network improves the performance of the hand-designed policy. In this way, we propose a way of using neural networks to improve expert designed controllers, while maintaining ease of understanding.
BibTeX
@inproceedings{li2019using,
  author = {Tianyu Li and Hartmut Geyer and Christopher G. Atkeson and Akshara Rai},
  title = {{Using Deep Reinforcement Learning to Learn High-Level Policies on the ATRIAS Biped}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=GUgiFlUqQU},
}
Figure