Learning to Sequence Robot Behaviors for Visual Navigation

Hadi Salman, Puneet Singhal, Tanmay Shankar, Peng Yin, Ali Salman, William Paivine, Guillaume Sartoretti, Matthew J. Travers, Howie ChosetarXiv 2018 · OpenReview

TL;DR

This paper introduces a hierarchical reinforcement learning framework that enables robots to learn and sequence low-level behaviors for visual navigation. By training a meta-policy to select appropriate low-level actions based on visual input, robots can navigate complex, previously unseen environments more effectively than relying on individual behaviors alone.

What Problem Does This Paper Address?

Autonomous mobile robots often need to navigate challenging and dynamic environments. While researchers have developed methods to learn low-level "behaviors" (like specific gaits or movement patterns) that allow robots to operate in certain conditions, effectively combining these behaviors to tackle new or complex scenarios remains a significant challenge. Traditional approaches often rely on hand-crafted rules or state representations that don't generalize well. This paper addresses the problem of how a robot can learn to sequence these pre-learned or learned low-level behaviors to achieve a task, specifically visual navigation, using only raw visual input.

Methods

The core of the proposed solution is a hierarchical reinforcement learning (RL) framework. This framework consists of two main components:

  1. Low-Level Control Policies: These are learned or designed behaviors that enable the robot to locomote and avoid obstacles within a specific, often elementary, environment. For example, a legged robot might have gaits for flat terrain versus steep inclines, and a wheeled robot might have policies for turning left or right. These policies operate on raw visual input.
  2. Meta-Level Policy: This higher-level policy acts as a "sequencer." Based on the robot's current visual observation, it actively selects which low-level policy to execute for a defined period (N time steps). The meta-policy is trained using Deep Q-Learning (DQN), a standard deep reinforcement learning algorithm.

The key innovation lies in training the meta-policy on compound environments that are combinations of the elementary environments where the low-level policies were trained. This forces the meta-policy to learn when to switch between behaviors to achieve optimal performance in novel, mixed environments.

Datasets

The paper does not explicitly mention specific datasets in the traditional sense. Instead, it uses simulated environments within the gym-gazebo framework, which is an extension of OpenAI Gym for robotics based on ROS and the Gazebo simulator. The robots are trained and evaluated in these simulated environments, which are constructed to represent:

  • Elementary Environments: Simple, visually distinct environments (e.g., rough terrain, maze-like corridors) used to train individual low-level policies.
  • Compound Environments: Combinations of these elementary environments, designed to test the meta-policy's ability to sequence behaviors.

Key Insights and Ideas

  • Learned Sequencing Outperforms Hand-Crafted: The meta-policy learned to sequence behaviors achieved significantly better results than a human-designed deterministic policy that manually assigned behaviors to specific environmental features. This highlights the power of learned adaptation.
  • Generalization Through Hierarchy: The hierarchical approach allows robots to generalize to new environments composed of familiar sub-components, even if they haven't encountered that specific combination during training.
  • Meta-Policy Agnosticism: The meta-policy is designed to be agnostic to the specific nature of the low-level behaviors, meaning it can sequence both learned and hand-crafted policies, offering flexibility.
  • Visual Input is Sufficient: The entire system relies solely on raw visual input from a monocular camera, demonstrating the capability of deep learning models to extract relevant information for complex decision-making.

Key Results

The paper demonstrates the effectiveness of their approach on two simulated robots:

  • Legged Robot: Successfully navigated a rough terrain with a high obstacle by learning to sequence two different gaits (one for speed, one for stepping).
  • Wheeled Robot (Turtlebot): Navigated a maze-like environment composed of two distinct elementary environments. The learned meta-policy achieved an average cumulative reward of 988.5 over 10 episodes, compared to 669.3 for a hand-crafted deterministic policy and significantly lower rewards for individual low-level policies (0% success rate for compound environments). The meta-policy also achieved an 80% success rate in trials, compared to 40% for the hand-crafted policy.

Limitations and Future Work

  • Limitations:
    • The low-level policies were trained on specific elementary environments and performed poorly on others, necessitating the meta-policy for generalization.
  • The paper primarily focuses on simulated environments.
  • Future Work:
    • Explore learning both the low-level and meta-level policies jointly, which could enable more adaptive and online learning capabilities.
  • Investigate the framework's applicability with other sensor modalities beyond cameras (e.g., LIDAR), as briefly tested in a dynamic environment scenario.

Summary updated

Abstract

Recent literature in the robotics community has focused on learning robot behaviors that abstract out lower-level details of robot control. To fully leverage the efficacy of such behaviors, it is necessary to select and sequence them to achieve a given task. In this paper, we present an approach to both learn and sequence robot behaviors, applied to the problem of visual navigation of mobile robots. We construct a layered representation of control policies composed of low- level behaviors and a meta-level policy. The low-level behaviors enable the robot to locomote in a particular environment while avoiding obstacles, and the meta-level policy actively selects the low-level behavior most appropriate for the current situation based purely on visual feedback. We demonstrate the effectiveness of our method on three simulated robot navigation tasks: a legged hexapod robot which must successfully traverse varying terrain, a wheeled robot which must navigate a maze-like course while avoiding obstacles, and finally a wheeled robot navigating in the presence of dynamic obstacles. We show that by learning control policies in a layered manner, we gain the ability to successfully traverse new compound environments composed of distinct sub-environments, and outperform both the low-level behaviors in their respective sub-environments, as well as a hand-crafted selection of low-level policies on these compound environments.
BibTeX
@misc{salman2018learning,
  author = {Hadi Salman and Puneet Singhal and Tanmay Shankar and Peng Yin and Ali Salman and William Paivine and Guillaume Sartoretti and Matthew J. Travers and Howie Choset},
  title = {{Learning to Sequence Robot Behaviors for Visual Navigation}},
  year = {2018},
  eprint = {1803.01446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.01446},
}
Figure