Learn to Navigate: Cooperative Path Planning for Unmanned Surface Vehicles Using Deep Reinforcement Learning

Xinyuan Zhou, Peng Wu, Haifeng Zhang, Weihong Guo, Yuanchang LiuIEEE Access 2019 · OpenReview

Abstract

Unmanned surface vehicles (USVs) have seen rapid growth and application in both military and civilian domains. Central to their operation is path planning—the technology ensuring navigation safety through collision-free trajectories. This work investigates Deep Reinforcement Learning (DRL) for USV path planning, focusing on reliable obstacle avoidance in constrained maritime environments. We present three sets of algorithms for single USV navigation and multi-USV formation maintenance, capable of robustly retaining or varying shapes to navigate cluttered obstacles. Validated in simulations and real-world harbor data, these DRL-based approaches offer a high-level artificial intelligence resolution to complex maritime navigation challenges.

Introduction

Autonomous navigation for USVs is traditionally handled by algorithms like A*, Artificial Potential Fields, or Fast Marching Methods. While effective, these conventional methods often require continuous map reconstruction—which can hinder real-time application—and rely on simplified kinematic models that fail to reflect actual vessel dynamics.

The complexity scales significantly with USV formations. Traditional formation control relies on rigid leader-follower mathematical models. In dense obstacle fields or narrow channels, a rigid shape can force follower vessels into collisions or necessitate inefficient detours. This research frames path planning as a model-free Markov Decision Process (MDP), enabling USVs to learn dynamic, kinematics-compliant navigation policies through trial and error.

System Architecture and Methodology

The proposed framework is structured into three hierarchical layers: the Task Management Layer, the Path Planning Layer, and the Task Execution Layer. This hierarchy allows for a separation of concerns between high-level mission assessment and low-level vehicle control.

Hierarchical control architecture for USV formation path planning, integrating task management, path planning, and execution layers.

The core of the system utilizes a Deep Q-Network (DQN) to approximate optimal action-value functions. The research develops three distinct algorithmic sets: single USV path planning, fixed-shape formation control, and flexible-shape formation control.

  • State & Action Spaces: The state space includes vessel position $(x, y)$ and heading $\theta$. To ensure physical executability, the action space is restricted to 13 discrete heading changes (ranging from $-60^\circ$ to $60^\circ$), reflecting the real-world constraints of a Springer USV.
  • Reward Shaping: For single USVs, rewards are granted for reaching goals, while penalties are applied for collisions, entering safety margins, or excessive heading changes to ensure smooth trajectories.
  • Formation Strategies: In multi-agent scenarios, the Fixed Shape paradigm penalizes deviations from a strict triangular formation. Conversely, the Flexible Shape paradigm relaxes these penalties near obstacles to prioritize collision avoidance over rigid geometry.

Experimental Setup

The algorithms were evaluated using simulated 2D binary maps with varying obstacle densities and practical maritime environments derived from real-world geographical data of Portsmouth and Plymouth harbors in the UK. These real-world maps provide realistic challenges such as narrow waterways and open-sea transitions.

Key Insights and Results

The Advantage of Flexibility

A critical finding is the performance gap between fixed and flexible formations. In dense obstacle fields, fixed-shape DQNs often failed due to heavy penalties for breaking formation. Conversely, the flexible-shape DQN learned "intelligent" behaviors: splitting or shrinking the formation to navigate tight spaces and autonomously re-forming into the target triangle in open water.

Speed and Safety Trade-offs

Training at different speeds (30 vs. 50 pixels per step) revealed that higher speeds resulted in shorter global trajectories but reduced safety margins by cutting closer to obstacles. Lower speeds produced more conservative, safer local paths.

Performance and Convergence

Single USV agents converged to optimal policies within 30,000 to 40,000 episodes. Complex 3-USV flexible formations required significantly more training (up to 1,000,000 episodes) due to the expanded state-action space, but successfully navigated complex real-world harbor simulations, such as the narrow channels of Portsmouth.

Conclusion and Future Work

This work demonstrates that DRL can effectively manage complex USV formation navigation in cluttered environments. However, current models assume calm waters. Future research will incorporate dynamic disturbances like wind, waves, and currents. Additionally, transitioning from discrete DQN actions to continuous-action algorithms like Deep Deterministic Policy Gradient (DDPG) or Soft Actor-Critic (SAC) could provide smoother, more direct autopilot control.

Summary updated

Abstract

Unmanned surface vehicle (USV) has witnessed a rapid growth in the recent decade and has been applied in various practical applications in both military and civilian domains. USVs can either be deployed as a single unit or multiple vehicles in a fleet to conduct ocean missions. Central to the control of USV and USV formations, path planning is the key technology that ensures the navigation safety by generating collision free trajectories. Compared with conventional path planning algorithms, the deep reinforcement learning (RL) based planning algorithms provides a new resolution by integrating a high-level artificial intelligence. This work investigates the application of deep reinforcement learning algorithms for USV and USV formation path planning with specific focus on a reliable obstacle avoidance in constrained maritime environments. For single USV planning, with the primary aim being to calculate a shortest collision avoiding path, the designed RL path planning algorithm is able to solve other complex issues such as the compliance with vehicle motion constraints. The USV formation maintenance algorithm is capable of calculating suitable paths for the formation and retain the formation shape robustly or vary shapes where necessary, which is promising to assist with the navigation in environments with cluttered obstacles. The developed three sets of algorithms are validated and tested in computer-based simulations and practical maritime environments extracted from real harbour areas in the UK.
BibTeX
@inproceedings{zhou2019learn,
  author = {Xinyuan Zhou and Peng Wu and Haifeng Zhang and Weihong Guo and Yuanchang Liu},
  title = {{Learn to Navigate: Cooperative Path Planning for Unmanned Surface Vehicles Using Deep Reinforcement Learning}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=_bzkAC1gZsm},
}
Figure