ReProHRL: Towards Multi-Goal Navigation in the Real World using Hierarchical Agents

Tejaswini Manjunath, Mozhgan Navardi, Prakhar Dixit, Bharat Prakash, Tinoosh MohseninarXiv 2023 · arXiv

Reinforcement Learning (RL) has long promised a future where robots learn complex tasks through trial and error. However, two major hurdles remain: the "Sim2Real" gap—where policies learned in pristine simulations fail in the messy real world—and the difficulty of learning long-horizon, multi-goal tasks. Researchers from the University of Maryland, Baltimore County (UMBC) have proposed a new framework, ReProHRL (Ready for Production Hierarchical RL), to tackle these challenges head-on.

TL;DR

ReProHRL is a hierarchical reinforcement learning framework that enables drones to navigate to multiple sequential goals in the real world by using object detection as a semantic bridge. By replacing raw image inputs with bounding box coordinates, the system drastically reduces the state space and achieves a 98% success rate in complex environments, significantly outperforming traditional end-to-end methods.

What Problem Does This Paper Address?

Training robots directly in the real world is slow, expensive, and dangerous. While simulators like AirSim provide a safe alternative, they suffer from a "visual gap": a policy trained on simulated pixels often gets confused by real-world lighting, textures, and shadows.

Furthermore, most RL agents struggle with "sparse rewards"—situations where the robot only gets a "good job" signal after completing a long sequence of actions. Navigating to one object is hard; navigating to three in a specific order is exponentially harder for a "flat" RL agent that tries to learn everything at once.

Methods

The authors break the problem down using a hierarchical approach and a clever pre-processing step to simplify what the robot "sees."

1. The Hierarchical Architecture

As shown in Figure 2, the system is split into two parts:

  • High-Level Controller: Acts as a state machine that decides which object the drone should go to next (the sub-goal).
  • Low-Level Policy: A Deep Q-Network (DQN) that takes the current sub-goal and sensor data to output primitive actions (left, right, straight).
Figure 2: Illustration of the architecture for a hierarchical agent used in this work. The High-level controller which acts as a state machine generates a sub-goal when the agent completes an episode. The Low-level Policy outputs an action at every step and the agent performs the action in the environment to reach the goals.

2. Object Detection as a Semantic Bridge

Instead of feeding raw camera images into the RL agent, ReProHRL uses YOLOv5 to detect objects and extract their bounding box (Bbox) coordinates.

  • Why this works: A bounding box representing a "red ball" looks identical to the RL agent whether it was generated from a high-fidelity simulation or a grainy real-world camera. This effectively "de-noises" the visual input and bridges the Sim2Real gap.
  • State Space Reduction: The agent's input is reduced from thousands of pixels to a simple 1x19 vector containing Bbox coordinates, a one-hot encoded goal, and lidar distance readings for collision avoidance (see Figure 3).
Figure 3: Details of Low-level Policy for goal-dependent navigation. The Bounding box (Bbox) coordinates from pre-processed image and goal vector (received from the High-level controller shown in Figure 2) are concatenated with the lidar embedding (used for collision avoidance) as an input to the low-level policy.

3. Curriculum Learning

To handle multiple goals, the authors used curriculum learning, progressively increasing the task difficulty from one goal to three during training.

Datasets

The researchers used a mix of simulated and real-world data:

  • Air-Learning: A simulator based on Unreal Engine and AirSim used to create high-fidelity indoor environments for training.
  • Custom Dataset: 4,000 manually annotated images (from both sim and real environments) were used to fine-tune the YOLOv5 object detector to ensure high accuracy on the specific objects used in the experiments.
  • Real-World Testing: Conducted in a 5mx5m room using a Crazyflie nano-drone equipped with an AI-deck for imaging and a Multi-ranger deck for lidar.

Key Insights and Ideas

  • Bboxes > Pixels: The most significant insight is that semantic representations (like bounding boxes) are far more robust for Sim2Real transfer than raw pixels. They provide a universal language that translates perfectly across domains.
  • Efficiency through Sparsity: By reducing the state space to just 19 values, the RL agent converges much faster. In multi-goal settings, ReProHRL reached near 100% success in just 22k steps, while baselines failed to reach that level even after 1 million steps.
  • Hardware Constraints Matter: The authors highlight that while the logic works, physical constraints like battery life are the next bottleneck. Adding processing decks to a nano-drone reduced its flight time from 7 minutes to just 3 minutes.

Key Results

The ReProHRL framework was tested against a state-of-the-art language-guided baseline.

  • Complex Environments: In environments with obstacles, ReProHRL achieved a 98% success rate, compared to just 80% for the baseline.
  • Multi-Goal Navigation: For two-goal tasks, ReProHRL hit 100% success rapidly, whereas the baseline struggled to generalize.
  • Real-World Deployment: The system was successfully deployed on a Crazyflie drone, proving that the hierarchical model and YOLO-based pre-processing are lightweight enough for edge computing on nano-UAVs.

Limitations and Future Work

While ReProHRL is a leap forward, the authors note several areas for improvement:

  • Fixed Sequences: The high-level controller currently follows a fixed sequence. Future versions could use a trainable high-level policy to dynamically choose the most efficient path between goals.
  • Energy Efficiency: The computational overhead of running YOLO and RL on-board is significant. Optimizing these models for even lower power consumption is critical for long-term autonomy.
  • Field of View: The agent is currently dependent on the object being within the camera's field of view to initiate navigation. Integrating a search behavior for when objects are hidden would make the system more robust.

Summary updated

Abstract

Robots have been successfully used to perform tasks with high precision. In real-world environments with sparse rewards and multiple goals, learning is still a major challenge and Reinforcement Learning (RL) algorithms fail to learn good policies. Training in simulation environments and then fine-tuning in the real world is a common approach. However, adapting to the real-world setting is a challenge. In this paper, we present a method named Ready for Production Hierarchical RL (ReProHRL) that divides tasks with hierarchical multi-goal navigation guided by reinforcement learning. We also use object detectors as a pre-processing step to learn multi-goal navigation and transfer it to the real world. Empirical results show that the proposed ReProHRL method outperforms the state-of-the-art baseline in simulation and real-world environments in terms of both training time and performance. Although both methods achieve a 100% success rate in a simple environment for single goal-based navigation, in a more complex environment and multi-goal setting, the proposed method outperforms the baseline by 18% and 5%, respectively. For the real-world implementation and proof of concept demonstration, we deploy the proposed method on a nano-drone named Crazyflie with a front camera to perform multi-goal navigation experiments.
BibTeX
@misc{manjunath2023reprohrl,
  author = {Tejaswini Manjunath and Mozhgan Navardi and Prakhar Dixit and Bharat Prakash and Tinoosh Mohsenin},
  title = {{ReProHRL: Towards Multi-Goal Navigation in the Real World using Hierarchical Agents}},
  year = {2023},
  eprint = {2308.08737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.08737},
}
Figure