VO-Safe Reinforcement Learning for Drone Navigation

Feiqiang Lin, Changyun Wei, Raphael Grech, Ze JiICRA 2024 · OpenReview

Abstract

This work is focused on reinforcement learning (RL)-based navigation for drones, whose localisation is based on visual odometry (VO). Such drones should avoid flying into areas with poor visual features, as this can lead to deteriorated localization or complete loss of tracking. To achieve this, we propose a hierarchical control scheme, which uses an RL-trained policy as the high-level controller to generate waypoints for the next control step and a low-level controller to guide the drone to reach subsequent waypoints. For the high-level policy training, unlike other RL-based navigation approaches, we incorporate awareness of VO performance into our policy by introducing pose estimation-related punishment. To aid robots in distinguishing between perception-friendly areas and unfavoured zones, we instead provide semantic scenes as input for decision-making instead of raw images. This approach also helps minimise the sim-to-real application gap.

Introduction

Autonomous drone navigation in GPS-denied environments typically relies on Visual Odometry (VO) to estimate movement by tracking visual features in camera frames. However, VO systems face significant challenges in "feature-poor" environments. Terrains such as open water, snow, or uniform grass lack the distinct landmarks required for stable tracking, often leading to localization failure. Furthermore, aggressive flight maneuvers common in end-to-end RL policies can induce motion blur or extreme camera tilts, further degrading VO performance.

Standard RL agents typically optimize for the shortest path to a goal, which often leads them directly into these hazardous zones. This research introduces VO-Safe RL, a framework designed to train agents that are "VO-aware," prioritizing path reliability over simple distance minimization.

Methodology

The proposed system utilizes a hierarchical control scheme to decouple high-level decision-making from low-level flight dynamics.

The hierarchical control architecture integrates semantic classification and visual odometry to inform an RL-based high-level policy.
  1. High-Level Controller: An RL agent trained using Proximal Policy Optimization (PPO). Instead of raw RGB images, it receives semantically segmented masks (identifying categories like "water" or "trees") and relative goal positions. This abstraction helps bridge the sim-to-real gap.
  2. Low-Level Controller: A traditional PID controller that translates RL-generated waypoints into stable motor commands, preventing the jittery movements that often cause VO tracking loss.
  3. VO-Aware Reward Function: During training in the Flightmare simulator, the agent receives a significant penalty if the VO system (ORB-SLAM2) fails. This teaches the agent that maintaining localization is a primary objective.

A key insight of this work is that the shortest path is not necessarily the optimal path when sensor reliability is considered. The VO-Safe agent learns to "hug" visually rich areas, such as shorelines, rather than crossing featureless expanses.

The VO-safe agent (green) learns to follow shorelines to maintain localization, whereas a standard agent (red) fails by attempting the shortest path over water.

By using semantic masks as a "universal language," the policy remains effective when transferred from simulation to real-world hardware. The agent recognizes the "meaning" of the terrain rather than being distracted by pixel-level differences in lighting or texture.

Experimental Results

The VO-Safe RL approach was evaluated against a standard state-based agent and an end-to-end VO-based baseline:

  • Success Rate: The proposed method achieved a 78% success rate, significantly outperforming the end-to-end baseline (56%), which frequently failed over water or during rapid turns.
  • Localization Accuracy: The system maintained a much lower Root Mean Square Error (RMSE) in position (0.38m) compared to the baseline (1.10m).
  • Real-World Validation: Physical tests confirmed that the drone could successfully navigate around featureless zones (e.g., white paper on a floor) by choosing curved paths over textured ground.

Conclusion

VO-Safe RL demonstrates that incorporating sensor-performance awareness into hierarchical reinforcement learning significantly improves drone navigation robustness. By prioritizing visually safe paths and utilizing semantic abstraction, the system effectively mitigates the risks of localization failure in feature-poor environments. Future work will explore the integration of obstacle avoidance and navigation in dynamic environments.

Summary updated

Abstract

This work is focused on reinforcement learning (RL)-based navigation for drones, whose localisation is based on visual odometry (VO). Such drones should avoid flying into areas with poor visual features, as this can lead to deteriorated localization or complete loss of tracking. To achieve this, we propose a hierarchical control scheme, which uses an RL-trained policy as the high-level controller to generate waypoints for the next control step and a low-level controller to guide the drone to reach subsequent waypoints. For the high-level policy training, unlike other RL-based navigation approaches, we incorporate awareness of VO performance into our policy by introducing pose estimation-related punishment. To aid robots in distinguishing between perception-friendly areas and unfavoured zones, we instead provide semantic scenes, as input for decision-making instead of raw images. This approach also helps minimise the sim-to-real application gap.
BibTeX
@inproceedings{lin2024vo,
  author = {Feiqiang Lin and Changyun Wei and Raphael Grech and Ze Ji},
  title = {{VO-Safe Reinforcement Learning for Drone Navigation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=w5E1SuCkMd},
}
Figure