AUV Path Planning Considering Ocean Current Disturbance Based on Cloud Desktop Technology

Shuai Xiao, Siyuan Hu, Shuai Xiao 0001, Jiachen Yang, Zuochen Zhang, Kunyu Zhang, Yong Zhu, Yubo ZhangSensors 2023 · DOI

TL;DR

This paper introduces the N-DDQNP, a novel deep reinforcement learning approach for Autonomous Underwater Vehicle (AUV) path planning that effectively handles dynamic ocean currents and obstacles. By integrating a noise network for improved exploration and prioritized experience replay for faster learning, N-DDQNP significantly outperforms existing methods in terms of path length and time. The research also leverages cloud desktop technology for real-time AUV monitoring and control, enhancing operational safety and efficiency.

What Problem Does This Paper Address?

The exploration and harvesting of ocean resources are increasingly vital, and Autonomous Underwater Vehicles (AUVs) are key tools for this. However, navigating the dynamic and unpredictable ocean environment, particularly dealing with ocean currents and obstacles, poses significant challenges for AUV path planning. Traditional methods often struggle with the complexity and real-time nature of these environments, leading to inefficient or unsafe navigation. Existing reinforcement learning approaches, like DQN, face issues with slow convergence and insufficient exploration, often getting stuck in local optima. Furthermore, the practical deployment of AUVs is hampered by the difficulty of real-time monitoring and control, requiring significant manpower for tasks like returning the AUV to its starting point after each mission.

Methods

The core of this paper is the N-DDQNP (Noise Network and Priority Experience Replay Dual DQN Network) model. This approach builds upon the Double Deep Q-Network (DDQN) framework and introduces two key innovations:

  • Noise Network: Instead of the traditional ε-greedy exploration strategy, N-DDQNP uses a noise network. This network adds parameterized noise to the learned weights, allowing for more adaptive and robust exploration during the initial training phases. This helps prevent the agent from getting trapped in local optima and improves the exploration-utilization balance.
  • Priority Experience Replay: To address the slow training speed and inefficient sampling of traditional DQN, the paper incorporates a prioritized experience replay mechanism. This mechanism samples experiences from the replay buffer based on their "priority," typically determined by their temporal difference (TD) error. This means more important or informative samples are sampled more frequently, leading to faster convergence and more stable training.

Additionally, the paper proposes a compound reward function that considers:

  • Distance: Euclidean distance to the target.
  • Ocean Current: The angle between the current and the AUV's heading, and the impact on speed.
  • Safety: A significant penalty for collisions with obstacles.

Finally, the research integrates Cloud Desktop Technology (specifically SPICE protocol and OpenStack) to enable remote observation and control of the AUV. This allows for real-time visualization of the AUV's navigation posture and the surrounding marine environment, facilitating safer operations and reducing the need for manual intervention.

Datasets

The experiments were conducted using real ocean current data from January and December 2018, sourced from the National Oceanic Data Center. This data was generated using the Princeton Ocean Model (POMgcs) and represents two distinct ocean current environments (C1 and C2). The experimental setup involved creating two-dimensional grid environments with varying numbers of obstacles and target points.

Key Insights and Ideas

  • Noise Network for Smarter Exploration: The noise network is a more sophisticated way to explore the state-action space than simple random actions. By learning to perturb its own weights, the network can adapt its exploration strategy, leading to better discovery of optimal paths.
  • Prioritized Learning for Efficiency: Focusing training on "surprising" or high-error experiences (via prioritized replay) is a powerful way to accelerate learning in complex environments, much like how humans learn faster from mistakes.
  • Holistic Reward Function is Crucial: Simply optimizing for distance isn't enough. Incorporating ocean currents and safety penalties into the reward function is essential for practical AUV navigation, balancing speed with survival.
  • Cloud Desktop Bridges the Gap: The integration of cloud desktop technology is a significant practical advancement. It transforms AUV operations from being physically constrained to being remotely manageable and observable, akin to controlling a drone from a laptop.

Key Results

The N-DDQNP model demonstrated superior performance across various experimental scenarios:

  • Path Length and Time: In the E1 environment, N-DDQNP achieved a path length 14.91% shorter and saved 14.83% of the time compared to DQN. Across other environments (E1-E8), N-DDQNP consistently outperformed traditional algorithms like I-RRT, IAPF, DQN, and DDQN in terms of path length and planning time. For instance, in the E5 environment, N-DDQNP was 20.65% faster than I-RRT.
  • Convergence Speed: The paper states that N-DDQNP exhibits higher convergence speed and average reward compared to DDQN and DQN.
  • Obstacle Avoidance: The N-DDQNP algorithm was successful in navigating complex environments with multiple obstacles, often generating smoother paths and maintaining a safer distance from obstacles compared to other methods. In some cases, algorithms like IAPF and DQN failed to reach the target due to collisions.
  • Cloud Desktop Utility: The cloud desktop setup enabled remote observation of the AUV's navigation posture and the marine environment, facilitating safer and more efficient exploration. It also allowed for remote control of the AUV, including its return path, reducing operational costs.

Limitations and Future Work

  • Training Speed: While N-DDQNP shows improved convergence, the paper acknowledges that training speed can still be a bottleneck.
  • Future Work: The authors suggest optimizing sampling efficiency through parallel training. This could involve constructing multiple sets of decision-making networks to calculate a total loss function and perform separate gradient updates, enabling faster exploration of high-value samples and further enhancing convergence speed.

Summary updated

BibTeX
@article{xiao2023auv,
  author = {Shuai Xiao and Siyuan Hu and Shuai Xiao and Jiachen Yang and Zuochen Zhang and Kunyu Zhang and Yong Zhu and Yubo Zhang},
  title = {{AUV Path Planning Considering Ocean Current Disturbance Based on Cloud Desktop Technology}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23177510},
}
Figure