Navigation in Restricted Channels Under Environmental Conditions: Fast-Time Simulation by Asynchronous Deep Reinforcement Learning

José Amendola, Lucas S. Miura, Anna Helena Reali Costa, Fábio Gagliardi Cozman, Eduardo Aoun TannuriIEEE Access 2020 · OpenReview

Abstract

This paper proposes an efficient method, based on reinforcement learning, to be used as ship controller in fast-time simulators within restricted channels. The controller must operate the rudder in a realistic manner in both time and angle variation so as to approximate human piloting. The method is well suited to scenarios where no previous navigation data is available; it takes into account, during training, both the effect of environmental conditions and also curves in channels. We resort to an asynchronous distributed version of the reinforcement learning algorithm Deep Q Network (DQN), handling channel segments as separate episodes and including curvature information as context variables. We tested our proposal in the channel of Porto Sudeste, in the southern Brazilian coast, with realistic environment scenarios where wind and current incidence varies along the channel. The method keeps a simple representation and can be applied to any port channel configuration that respects local technical regulations.

Introduction: The Challenge of Restricted Water Navigation

Navigating a 160,000-ton Suezmax tanker through a narrow port channel is a high-stakes engineering challenge. In these "restricted waters," the proximity of the seafloor and channel banks creates complex hydrodynamic interactions that significantly alter ship handling compared to open-sea navigation. Human pilots must constantly compensate for these forces, alongside unpredictable wind and tidal currents, to keep the vessel safely within the channel margins.

Traditionally, fast-time simulations—used by port engineers for risk assessment and infrastructure design—have relied on simplified autopilot algorithms. However, these systems often produce "perfect" trajectories that do not reflect the realities of human piloting, such as the need to anticipate curves or maintain a drift angle against cross-currents. This study addresses this gap by developing a ship controller using Asynchronous Deep Reinforcement Learning that mimics the nuanced decision-making of a human pilot.

Methodology: Asynchronous Deep Reinforcement Learning

The core of the proposed controller is the APE-X-DQN (Asynchronous Prioritized Experience Replay Deep Q-Network) architecture. This approach was chosen for its efficiency in handling complex, continuous environments.

APE-X Architecture

To accelerate the learning process, the researchers utilized a distributed architecture. Multiple "actors" run in parallel across 48 CPU cores, each interacting with its own instance of the simulator to collect experiences. These experiences are stored in a centralized prioritized replay buffer, from which a "learner" pulls data to update the global neural network. This parallelization allowed the team to collect over 6 million experience steps and train a successful policy in under 90 minutes.

Task Segmentation and Contextual Awareness

Learning to navigate a 16km channel in a single episode is computationally expensive and prone to failure. To solve this, the authors implemented Task Segmentation, breaking the channel into overlapping segments. Furthermore, to help the agent "look ahead," the angle of the next channel segment was included as a context variable in the Q-function. This allows the AI to distinguish between a straight path and the approach to a sharp turn, enabling it to initiate maneuvers before reaching the curve—a key trait of experienced human pilots.

Discrete Control Logic

To ensure realistic behavior, the AI does not have infinite precision. Instead, it selects from five discrete rudder levels (e.g., -50%, -20%, 0%, 20%, 50% of maximum angle) at 10-second intervals. This prevents the high-frequency "chatter" or erratic corrections often seen in standard automated controllers and better approximates the physical limitations and habits of human operators.

Experimental Setup: High-Fidelity Simulation

The agent was trained and tested within the TPN-USP simulator, a high-fidelity environment that utilizes complex 6-DOF (degrees of freedom) equations to model ship dynamics.

The TPN-USP high-fidelity maritime simulator environment used for training and testing the RL agent.

The study focused on a digital twin of the Porto Sudeste Access Channel in Brazil. The 16km path includes three straight sections and a challenging 1600m radius bend. The vessel modeled was a fully loaded Suezmax DP tanker (160,000 DWT). Training involved four primary environmental scenarios, combining Flood and Ebb tides (0.4 to 1.0 knots) with West and North winds (20 knots), ensuring the agent learned to handle varying lateral forces.

Results and Discussion

The APE-X-DQN controller demonstrated significant improvements over traditional methods and other RL algorithms like PPO (Proximal Policy Optimization).

  • Safety and Precision: In testing, the AI-controlled ship maintained a minimum distance of 64 meters from the channel margins. In maritime engineering, maintaining a buffer of more than half a "beam" (ship width) is considered a safe and successful maneuver.
  • Strategic Maneuvering: The agent successfully learned to maintain a "drift angle"—navigating slightly sideways—to counteract lateral forces from wind and current. This allowed it to stay on the centerline without constant, erratic rudder adjustments.
  • Robustness to Unseen Conditions: When tested on "Scenario E10"—a critical environmental condition not included in the training set—the agent successfully navigated 95% of the channel on its first attempt, demonstrating strong generalization capabilities.
  • Convergence Efficiency: The asynchronous approach reached a stable, successful policy faster and with less variance than centralized RL methods.

Conclusion and Future Directions

This research provides a robust framework for creating realistic ship controllers in fast-time simulators. By combining distributed reinforcement learning with contextual awareness of channel geometry, the AI achieves human-like performance in complex environmental conditions.

Future work will aim to expand the controller's capabilities beyond rudder control to include engine speed adjustments and the use of tugboats. Additionally, the researchers plan to incorporate dynamic environmental factors, such as "two-way" traffic and the hydrodynamic suction effects caused by passing other large vessels, and eventually tackle the final, low-speed berthing phase of navigation.

Summary updated

Abstract

This paper proposes an efficient method, based on reinforcement learning, to be used as ship controller in fast-time simulators within restricted channels. The controller must operate the rudder in a realistic manner in both time and angle variation so as to approximate human piloting. The method is well suited to scenarios where no previous navigation data is available; it takes into account, during training, both the effect of environmental conditions and also curves in channels. We resort to an asynchronous distributed version of the reinforcement learning algorithm Deep Q Network (DQN), handling channel segments as separate episodes and including curvature information as context variables (thus moving away from most work in the literature). We tested our proposal in the channel of Porto Sudeste, in the southern Brazilian coast, with realistic environment scenarios where wind and current incidence varies along the channel. The method keeps a simple representation and can be applied to any port channel configuration that respects local technical regulations.
BibTeX
@inproceedings{amendola2020navigation,
  author = {José Amendola and Lucas S. Miura and Anna Helena Reali Costa and Fábio Gagliardi Cozman and Eduardo Aoun Tannuri},
  title = {{Navigation in Restricted Channels Under Environmental Conditions: Fast-Time Simulation by Asynchronous Deep Reinforcement Learning}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=rcfrelsGFR},
}
Figure