Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Mohammad Etemad, Nader Zare, Mahtab Sarvmaili, Amílcar Soares Júnior, Bruno Brandoli Machado, Stan Matwin

TL;DR

This paper introduces a novel deep reinforcement learning (DRL) approach, the Vessel Navigator with Planner and Local View (VNPLV), for autonomous vessel navigation in unknown 2D environments. VNPLV combines long-term path planning with short-term DRL-based obstacle avoidance, significantly outperforming a baseline method in terms of successful navigation, especially for longer distances.

What Problem Does This Paper Address?

Autonomous navigation for Unmanned Surface Vehicles (USVs) is a critical area of research, aiming to enable vessels to perform missions safely and efficiently without human intervention. A core challenge in this domain is path planning and obstacle avoidance, particularly in unknown or dynamic environments where the vessel cannot rely on pre-existing maps or predictable conditions. Traditional methods often struggle with the complexity and uncertainty of real-world maritime scenarios. While deep reinforcement learning (DRL) has shown promise, issues like instability and divergence can arise when combining off-policy learning with function approximation. This paper tackles the problem of developing a robust DRL-based navigation system that can effectively plan paths and avoid obstacles in dynamic, unknown 2D maritime environments.

Methods

The VNPLV method is built upon a DRL framework and incorporates two key novel components:

  • Rolling Wave Planning: Inspired by agile methodologies, this approach combines long-term and short-term planning.
  • Long-Term Planning (Path Planner): A Floyd algorithm is used to generate an initial, optimal path from origin to destination, assuming no obstacles. This provides a general direction.
  • Short-Term Decision-Making (DRL Agent): A deep Q-learning agent, utilizing a local view of the environment, handles immediate obstacle avoidance and fine-tunes the path towards intermediate goals derived from the long-term plan.
  • Local View Strategy: Instead of relying on a global view of the environment, the agent's decision-making is based on a limited, local observation window. This reduces the state space, making the learning process more manageable and potentially more stable.
  • Hierarchical Approach: The overall strategy involves:
  • Generating a global path using Floyd's algorithm.
  • Simplifying this path using the Ramer-Douglas-Peucker algorithm to create a sequence of intermediary goals.
  • Using the DRL agent with a local view to navigate towards the current intermediary goal, dynamically avoiding obstacles. Once an intermediary goal is reached, the agent shifts its focus to the next one.

The paper also details a Vanilla Vessel Navigator (VVN) as a baseline, which is an implementation of a DRL approach without the planner and local view enhancements.

Datasets

The authors created a simulated 2D maritime environment based on public earth elevation data for Halifax, Nova Scotia, Canada. This environment included:

  • A water layer and a land layer.
  • A 50-meter buffer around the land, defining navigable areas.
  • Dynamically generated moving objects acting as obstacles within the environment during episodes.

No specific pre-existing datasets were used for training; the environment itself was generated for simulation purposes.

Key Insights and Ideas

  • The Synergy of Planning and Reinforcement Learning: Combining a traditional path planner for global direction with a DRL agent for local, reactive obstacle avoidance proves to be more effective than either approach alone. The planner provides a robust, obstacle-free route, while the DRL agent handles the unpredictable real-time challenges.
  • Local View for Efficiency and Stability: Restricting the agent's observation to a local view significantly reduces the complexity of the state space. This is crucial for improving the stability and efficiency of DRL algorithms, which can struggle with high-dimensional inputs.
  • Rolling Wave Planning for Adaptability: The agile-inspired rolling wave planning allows the agent to adapt its short-term actions based on immediate environmental feedback while still having a long-term objective. This is particularly useful in dynamic environments.
  • Overcoming VVN's Limitations: The baseline VVN struggles with longer distances due to an increasing search space and a higher probability of getting stuck in repetitive actions. VNPLV directly addresses this by providing a more structured approach to navigation.

Key Results

  • Significant Performance Improvement: The VNPLV method demonstrated a substantial improvement in performance compared to the VVN baseline.
  • Enhanced Success Rate: For long-distance missions (0.32 degrees), VNPLV achieved a mean Rate of Arrival to Destination (RATD) of 79.44%, while VVN only managed 24.13%. This represents a 55.31% average improvement for long-distance missions.
  • Stability Across Distances: VNPLV showed consistent performance across various distances, whereas VVN's performance degraded significantly as the distance between the origin and destination increased.
  • Statistical Significance: A T-test confirmed a statistically significant difference in performance between VVN and VNPLV, with VNPLV being superior.

Limitations and Future Work

  • Limitations:
    • The current work uses a simulated environment. The dynamic obstacles are generated randomly, not based on real-world traffic patterns.
  • The VVN baseline, while improved, still has limitations in handling long-distance navigation.
  • Future Work:
    • Real-world Data Integration: Connect the dynamic obstacle generation to real-world AIS (Automatic Identification System) messages from vessels to create more realistic and challenging scenarios.
  • Advanced Neural Network Architectures: Explore sequence model networks, such as RNNs and CNNs, to incorporate memory of past agent actions for improved decision-making.
  • Multi-Agent Scenarios: Introduce other agents into the environment and define global tasks that require simultaneous execution by multiple agents.
  • Broader Applicability: Extend the adaptive planning methodology to other domains beyond autonomous navigation.

Summary updated

Abstract

Unmanned Surface Vehicles technology (USVs) is an exciting topic that essentially deploys an algorithm to safely and efficiently performs a mission. Although reinforcement learning is a well-known approach to modeling such a task, instability and divergence may occur when combining off-policy and function approximation. In this work, we used deep reinforcement learning combining Q-learning with a neural representation to avoid instability. Our methodology uses deep q-learning and combines it with a rolling wave planning approach on agile methodology. Our method contains two critical parts in order to perform missions in an unknown environment. The first is a path planner that is responsible for generating a potential effective path to a destination without considering the details of the root. The latter is a decision-making module that is responsible for short-term decisions on avoiding obstacles during the near future steps of USV exploitation within the context of the value function. Simulations were performed using two algorithms: a basic vanilla vessel navigator (VVN) as a baseline and an improved one for the vessel navigator with a planner and local view (VNPLV). Experimental results show that the proposed method enhanced the performance of VVN by 55.31 on average for long-distance missions. Our model successfully demonstrated obstacle avoidance by means of deep reinforcement learning using planning adaptive paths in unknown environments.
BibTeX
@inproceedings{etemad2020using,
  author = {Mohammad Etemad and Nader Zare and Mahtab Sarvmaili and Amílcar Soares Júnior and Bruno Brandoli Machado and Stan Matwin},
  title = {{Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments}},
  booktitle = {Canadian ai},
  year = {2020},
  doi = {10.1007/978-3-030-47358-7\_21},
}
Figure