Hierarchies of Planning and Reinforcement Learning for Robot Navigation

Jan Wöhlke, Felix Schmitt, Herke van Hoof

Abstract: Solving robotic navigation tasks via reinforcement learning (RL) is challenging due to their sparse reward and long decision horizon nature. However, in many navigation tasks, high-level (HL) task representations, like a rough floor plan, are available. Previous work has demonstrated efficient learning by hierarchal approaches consisting of path planning in the HL representation and using sub-goals derived from the plan to guide the RL policy in the source task. However, these approaches usually neglect the complex dynamics and sub-optimal sub-goal-reaching capabilities of the robot during planning. This work overcomes these limitations by proposing a novel hierarchical framework that utilizes a trainable planning policy for the HL representation. Thereby robot capabilities and environment conditions can be learned utilizing collected rollout data. We specifically introduce a planning policy based on value iteration with a learned transition model (VI-RL). In simulated robotic navigation tasks, VI-RL results in consistent strong improvement over vanilla RL, is on par with vanilla hierarchal RL on single layouts but more broadly applicable to multiple layouts, and is on par with trainable HL path planning baselines except for a parking task with difficult non-holonomic dynamics where it shows marked improvements.

Introduction

Reinforcement Learning (RL) is a powerful tool for robotic control, but it faces significant hurdles in navigation tasks. The primary challenge is the "sparse reward" problem: a robot may receive no feedback for thousands of steps until it finally reaches its destination. In large, complex environments, a standard RL agent might never encounter the goal through random exploration.

While high-level information like floor plans or topological maps is often available, traditional hierarchical RL (HRL) methods frequently fail to bridge the gap between abstract planning and physical reality. A shortest-path planner might suggest a route that is physically impossible for a specific robot—such as a car-like vehicle attempting a turn that is too sharp—or lead a robot through terrain that is slippery or slow. This paper introduces VI-RL, a framework designed to close this loop by learning the robot's actual capabilities and incorporating them into the high-level planning process.

Methodology: The VI-RL Framework

The proposed system utilizes a two-layer hierarchy to manage the complexity of navigation:

  1. High-Level (HL) Planner: This component operates on a discrete grid representation of the environment. Instead of relying on a static shortest-path algorithm, it employs Value Iteration (VI) with a learned transition model. By observing the success and failure of the low-level agent, the HL planner updates its beliefs about which movements are feasible. If a certain transition is difficult or slow, the planner learns to route around it.
  2. Low-Level (LL) Policy: This is a goal-conditioned RL agent (trained using TRPO) that manages continuous control, such as motor torques. It receives intrinsic rewards for reaching the specific sub-goals provided by the HL planner.
The VI-RL hierarchical framework: The High-Level policy generates sub-goals based on a learned transition model, while the Low-Level RL policy executes actions and provides rollout data back to the planner.

The core innovation of VI-RL is its ability to learn the probability of success for every transition. Unlike "optimistic" planners that assume all adjacent cells are reachable, VI-RL adapts to the robot's dynamics and the environment's specific constraints.

Experimental Setup

The researchers evaluated VI-RL across three distinct simulated environments, each testing different aspects of the navigation problem:

  • Four Rooms: A point-mass robot navigating a grid where one room contains "hard-to-traverse" terrain that significantly slows movement.
  • MuJoCo Ant Mazes: A complex quadruped "ant" robot navigating 25 different randomly generated maze layouts, testing the system's ability to generalize.
  • Vehicle Parking: A car-like (non-holonomic) robot that must reach a specific spot with a precise orientation. This is highly challenging because the robot cannot move sideways and has a constrained turning radius.
The three evaluation environments: (a) Four Rooms with varied terrain, (b) MuJoCo Ant Mazes with complex dynamics, and (c) Vehicle Parking with non-holonomic constraints.

Key Insights and Results

Dynamics-Aware Planning

A major finding is that shortest-path planning is often suboptimal when robot capabilities are non-uniform. In the "Four Rooms" task, an optimistic planner consistently failed by attempting to cross slow terrain and timing out. VI-RL, however, learned that this path was "expensive" in terms of time and successfully adapted to take longer but faster routes.

Closing the Feedback Loop

Most hierarchical systems are top-down, where the planner dictates and the RL agent follows. VI-RL creates a feedback loop where the RL agent's performance data teaches the planner what is actually possible. This proved critical in the Vehicle Parking task. While other state-of-the-art HRL methods (like HiRO) struggled to reach a 40% success rate, VI-RL achieved over 80% success by explicitly learning the physical transition constraints of the vehicle.

Performance Summary

  • Superiority over Vanilla RL: VI-RL reached goals much faster and more reliably than "flat" RL, which often failed to learn entirely due to sparse rewards.
  • Generalization: The system demonstrated strong performance across multiple maze layouts in the Ant Maze environment, showing it is more broadly applicable than non-adaptive hierarchical methods.

Conclusion

VI-RL demonstrates that integrating a learned transition model into high-level planning significantly improves the robustness of robotic navigation. By acknowledging that the "shortest" path on a map isn't always the "best" path for a specific robot, the framework bridges the gap between abstract planning and physical execution.

Future work aims to remove the reliance on pre-defined high-level state spaces by learning to automatically discretize environments from raw sensor data. Additionally, testing in real-world environments with sensor noise and uncertainty remains a key next step for the research.

Summary updated

Abstract

Solving robotic navigation tasks via reinforcement learning (RL) is challenging due to their sparse reward and long decision horizon nature. However, in many navigation tasks, high-level (HL) task representations, like a rough floor plan, are available. Previous work has demonstrated efficient learning by hierarchal approaches consisting of path planning in the HL representation and using sub-goals derived from the plan to guide the RL policy in the source task. However, these approaches usually neglect the complex dynamics and sub-optimal sub-goal-reaching capabilities of the robot during planning. This work overcomes these limitations by proposing a novel hierarchical framework that utilizes a trainable planning policy for the HL representation. Thereby robot capabilities and environment conditions can be learned utilizing collected rollout data. We specifically introduce a planning policy based on value iteration with a learned transition model (VI-RL). In simulated robotic navigation tasks, VI-RL results in consistent strong improvement over vanilla RL, is on par with vanilla hierarchal RL on single layouts but more broadly applicable to multiple layouts, and is on par with trainable HL path planning baselines except for a parking task with difficult non-holonomic dynamics where it shows marked improvements.
BibTeX
@inproceedings{wohlke2021hierarchies,
  author = {Jan Wöhlke and Felix Schmitt and Herke van Hoof},
  title = {{Hierarchies of Planning and Reinforcement Learning for Robot Navigation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=TXVLWTsgx4b},
}
Figure