Continuous Control with Deep Reinforcement Learning for Autonomous Vessels

Nader Zare, Bruno Brandoli, Mahtab Sarvmaili, Amílcar Soares, Stan MatwinarXiv 2021 · arXiv

Navigating maritime vessels through complex environments requires balancing goal-oriented movement with robust obstacle avoidance. While Deep Reinforcement Learning (DRL) has shown success in robotics, its application to autonomous ships often suffers from poor generalization—an agent trained in one harbor may fail in another due to overfitting to specific geographical geometries. This paper introduces the Continuous Vessel Navigator (CVN), a framework designed to bridge the gap between simulation and real-world versatility through continuous control and a novel data augmentation strategy.

The Generalization Challenge

Traditional maritime navigation models often rely on discrete action spaces (e.g., fixed turning increments), which lack the precision required for smooth vessel control. More importantly, these models struggle to generalize to unseen maps. To address this, the researchers propose a system that combines the Deep Deterministic Policy Gradient (DDPG) algorithm with a focus on local environmental features rather than global map memorization.

Methodology: The CVN Framework

Local View Maker

Instead of processing a massive global map, the agent utilizes a Local View Maker. This component generates a 51x51 pixel local representation centered on the vessel. By focusing on immediate surroundings—identifying land (brown pixels) and the target (green pixels)—the agent learns universal navigation principles rather than specific routes.

The Local View Maker provides a 51x51 pixel window centered on the vessel to focus the agent's attention on immediate obstacles and the goal.

State-Action Rotation (SAR)

The core innovation of this work is State-Action Rotation (SAR). This data augmentation technique leverages the inherent symmetry of navigation. When the agent experiences a transition (State $\rightarrow$ Action $\rightarrow$ New State), the system rotates that experience by 90°, 180°, and 270°, storing all four versions in the replay buffer. This teaches the agent that the logic of avoiding an obstacle remains constant regardless of the ship's orientation.

The State-Action Rotation (SAR) process generates augmented training data by rotating local views and corresponding actions, improving the agent's ability to generalize to new environments.

Path Simplification with LARDP

To guide the DRL agent, the system uses a modified path-simplification algorithm called Land Attended Rammer-Douglas-Peucker (LARDP). Standard algorithms often place waypoints on land when simplifying a complex path. LARDP ensures that intermediate goals remain in navigable water, preventing the agent from being led into "trap" areas near shorelines.

Key Results and Insights

The researchers evaluated the CVN model by training it on geographical data from Halifax and testing it on the unseen environment of Montreal. The results demonstrated significant performance gains:

  • Generalization Success: The CVN model with SAR achieved a 30.82% improvement in the Rate of Arrival To Destination (RATD) on the unseen Montreal map compared to baseline models.
  • Consistency: SAR consistently improved success rates by nearly 12% across various scenarios, proving that symmetry-based augmentation effectively reduces overfitting.
  • Training Dynamics: Interestingly, the study found that training with larger distances between start and end points improved performance. Longer trips forced the agent to navigate complex straits more frequently, providing richer training data.

Conclusion

By combining continuous control with symmetry-aware data augmentation, the CVN framework significantly enhances the ability of autonomous vessels to navigate unfamiliar maritime environments. Future work will focus on integrating dynamic collision avoidance (COLREGs) and transitioning from map-based inputs to real-time sensor fusion involving LiDAR and camera feeds.

Summary updated

Abstract

Maritime autonomous transportation has played a crucial role in the globalization of the world economy. Deep Reinforcement Learning (DRL) has been applied to automatic path planning to simulate vessel collision avoidance situations in open seas. End-to-end approaches that learn complex mappings directly from the input have poor generalization to reach the targets in different environments. In this work, we present a new strategy called state-action rotation to improve agent's performance in unseen situations by rotating the obtained experience (state-action-state) and preserving them in the replay buffer. We designed our model based on Deep Deterministic Policy Gradient, local view maker, and planner. Our agent uses two deep Convolutional Neural Networks to estimate the policy and action-value functions. The proposed model was exhaustively trained and tested in maritime scenarios with real maps from cities such as Montreal and Halifax. Experimental results show that the state-action rotation on top of the CVN consistently improves the rate of arrival to a destination (RATD) by up 11.96% with respect to the Vessel Navigator with Planner and Local View (VNPLV), as well as it achieves superior performance in unseen mappings by up 30.82%. Our proposed approach exhibits advantages in terms of robustness when tested in a new environment, supporting the idea that generalization can be achieved by using state-action rotation.
BibTeX
@misc{zare2021continuous,
  author = {Nader Zare and Bruno Brandoli and Mahtab Sarvmaili and Amílcar Soares and Stan Matwin},
  title = {{Continuous Control with Deep Reinforcement Learning for Autonomous Vessels}},
  year = {2021},
  eprint = {2106.14130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.14130},
}
Figure