Deep Reinforcement Learning to Acquire Navigation Skills for Wheel-Legged Robots in Complex Environments

Xi Chen, Ali Ghadirzadeh, John Folkesson, Patric JensfeltIROS 2018 · OpenReview

TL;DR

This paper tackles the challenge of teaching complex navigation skills to wheel-legged robots using deep reinforcement learning. They propose a novel approach that breaks down the navigation task into smaller, manageable behaviors and uses domain randomization to improve the robot's ability to focus on relevant sensory information, leading to significantly better data efficiency and success rates.

What Problem Does This Paper Address?

Navigating complex and dynamic environments is a significant hurdle for mobile robots. Traditional reinforcement learning (RL) methods struggle with this due to several inherent challenges:

  • Reward Sparsity: Rewards are often infrequent, making it hard for the agent to learn which actions lead to success.
  • Temporal Complexity: The long-term consequences of actions are difficult to attribute to specific decisions.
  • High-Dimensional Sensorimotor Spaces: The sheer amount of sensory input and the complexity of motor commands create a vast space for the agent to explore.

These issues make it difficult for RL agents to efficiently learn robust navigation policies, especially for robots with complex locomotion capabilities like wheel-legged robots, which can perform intricate maneuvers such as adjusting their body height or squeezing through narrow spaces.

Methods

The authors introduce a two-pronged approach to address these challenges:

  1. Behavior Decomposition: Instead of training a single, monolithic policy for the entire navigation task, they break it down into several simpler, manageable "behaviors." These include basic movements like moving straight, navigating around obstacles, and more complex maneuvers like lifting the body to overcome high obstacles or lowering it to pass through narrow corridors. Secondary policies are trained for each of these behaviors in simplified environments. The knowledge gained from these specialized policies is then used to train a primary, general-purpose navigation policy. This decomposition helps mitigate reward sparsity and credit assignment problems.
  2. Domain Randomization: To improve the robot's ability to generalize and focus on task-relevant sensory information (like the position of upcoming obstacles), they employ domain randomization. This technique involves training the policy in simulated environments where non-essential aspects of the scene (e.g., obstacle appearance, positions, and number) are randomized. Crucially, this randomization is applied in a way that doesn't invalidate the learned behaviors from the simpler environments. By replaying learned actions in these randomized environments, they generate a rich dataset of successful trajectories that helps the primary policy learn to attend to important cues and ignore irrelevant ones.

The core learning algorithm used is an actor-critic framework, optimized using Proximal Policy Optimization (PPO), which is a state-of-the-art policy gradient method. The network architecture takes a height-map image observation, along with robot and target poses, and outputs motor commands.

Datasets

The paper does not explicitly mention or introduce new public datasets. Instead, they generate their own training data through simulations.

  • Secondary Policies: Trained in simplified environments tailored to specific behaviors.
  • Primary Policy: Trained using a combination of real interactions with complex environments and a large batch of pre-generated trajectories derived from domain randomization. This batch consists of approximately 5 million trajectories.

Key Insights and Ideas

  • Decomposition is Key: Breaking down a complex task into simpler, learnable sub-skills significantly improves RL agent performance, especially in environments with sparse rewards.
  • Domain Randomization for Focus: Randomizing non-essential aspects of the environment forces the policy to learn more robust representations that are less sensitive to superficial details, leading to better generalization and task-relevant attention.
  • Leveraging Past Successes: Using pre-generated successful trajectories from domain randomization can effectively "bootstrap" the training of the primary policy, overcoming the initial data inefficiency and reward sparsity issues inherent in RL.
  • Human-like Skill Acquisition: The approach of learning individual skills and then combining them mirrors how humans learn complex activities, suggesting a promising direction for robot learning.

Key Results

The proposed method demonstrates significant improvements over a baseline PPO approach trained without the domain randomization batch:

  • Success Rate: A nearly 20% increase in success rate for the primary policy.
  • Data Efficiency: The approach is more data-efficient, requiring fewer training steps to achieve high performance.
  • Trajectory Quality: The generated trajectories are shorter (10 steps shorter on average) and more collision-free (80% success rate compared to 55% for the baseline).
  • Task-Relevant Attention: Experiments show that the trained model can identify and attend to relevant obstacles, ignoring those that have been removed from the scene, indicating a better understanding of the environment.

Limitations and Future Work

  • Limitations: The paper acknowledges the data-inefficiency of RL, which their method aims to mitigate. While domain randomization helps, it's still a simulation-based approach, and transferring to the real world can present further challenges. The current work focuses on navigation; more complex dynamic tasks are not addressed.
  • Future Work: The authors plan to:
  • Apply their domain randomization approach without further interactive RL training.
  • Extend their method to real robots for dynamic tasks like driving on uneven terrain.
  • Incorporate more complex skills such as walking or climbing, which require higher-dimensional action spaces.

Summary updated

Abstract

Mobile robot navigation in complex and dynamic environments is a challenging but important problem. Reinforcement learning approaches fail to solve these tasks efficiently due to reward sparsities, temporal complexities and high-dimensionality of sensorimotor spaces which are inherent in such problems. We present a novel approach to train action policies to acquire navigation skills for wheel-legged robots using deep reinforcement learning. The policy maps height-map image observations to motor commands to navigate to a target position while avoiding obstacles. We propose to acquire the multifaceted navigation skill by learning and exploiting a number of manageable navigation behaviors. We also introduce a domain randomization technique to improve the versatility of the training samples. We demonstrate experimentally a significant improvement in terms of data-efficiency, success rate, robustness against irrelevant sensory data, and also the quality of the maneuver skills.
BibTeX
@inproceedings{chen2018deep,
  author = {Xi Chen and Ali Ghadirzadeh and John Folkesson and Patric Jensfelt},
  title = {{Deep Reinforcement Learning to Acquire Navigation Skills for Wheel-Legged Robots in Complex Environments}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=eN5xRRElqcx},
}
Figure