Three-Dimensional Dynamic Trajectory Planning for Autonomous Underwater Robots Under the PPO-IIFDS Framework

Liqiang Liu, Min Sun, Enjiao Zhao, Kun ZhuJournal of Marine Science and Engineering 2025 · DOI

TL;DR

This paper introduces a novel framework for 3D dynamic trajectory planning in Autonomous Underwater Vehicles (AUVs) by integrating Proximal Policy Optimization (PPO) with an Improved Interfered Fluid Dynamic System (IIFDS). This hybrid approach allows AUVs to generate smooth, energy-efficient, and collision-free paths in complex, dynamic underwater environments by dynamically optimizing planning parameters and incorporating crucial AUV constraints.

What Problem Does This Paper Address?

Autonomous Underwater Vehicles (AUVs) are increasingly vital for marine exploration and monitoring. However, navigating the complex and dynamic underwater environment poses significant challenges for trajectory planning. These challenges include:

  • Dynamic Obstacles: The presence of moving obstacles like marine life, ice floes, or other underwater vehicles requires real-time adaptation.
  • Environmental Complexity: Three-dimensional terrains, variable ocean currents, and unpredictable conditions add layers of difficulty.
  • Balancing Objectives: Trajectory planning must simultaneously consider safety (obstacle avoidance), efficiency (reaching the target quickly and with minimal energy), and feasibility (adhering to the AUV's physical limitations).
  • Computational Demands: Real-time planning in complex environments requires efficient algorithms.

Traditional methods like graph search, heuristic algorithms (e.g., Artificial Potential Fields), and even some intelligent algorithms struggle to effectively balance these competing demands, often leading to suboptimal, unsafe, or computationally expensive trajectories.

Methods

The core of this work is a novel hybrid framework that combines two key components:

  • Improved Interfered Fluid Dynamic System (IIFDS): This acts as the "planning layer." Inspired by how fluids navigate around obstacles, IIFDS generates obstacle-adaptive trajectories by dynamically adjusting flow field parameters. The "improved" aspect involves incorporating tangential vectors to allow for more arbitrary movement around obstacles, preventing stagnation. Crucially, this version also integrates AUV-specific dynamics and kinematics constraints directly into the planning process, ensuring generated paths are physically executable.
  • Proximal Policy Optimization (PPO): This deep reinforcement learning algorithm serves as the "learning and decision-making layer." PPO is used to optimize the critical parameters within the IIFDS (repulsion, tangential, and directional coefficients). This is a significant improvement over traditional IIFDS, which relied on fixed parameters. PPO learns to adjust these parameters dynamically based on environmental feedback.

Key Innovations:

  • Multi-Objective Reward Function: The PPO reward function is significantly enhanced to include five objectives: obstacle avoidance, target distance, trajectory smoothness, dynamics constraints, and energy efficiency. This addresses the "sparse reward" problem common in RL and provides more nuanced guidance.
  • Integrated Dynamics and Kinematics Constraints: Unlike previous IIFDS approaches, this framework explicitly incorporates AUV motion constraints into the planning layer, ensuring generated trajectories are feasible.
  • PPO-IIFDS Integration: The synergy between PPO's learning capabilities and IIFDS's planning mechanism allows for adaptive, efficient, and robust trajectory generation in complex dynamic environments.

Datasets

The paper does not explicitly mention using pre-existing datasets. Instead, it focuses on constructing a novel, diverse, and dynamically complex obstacle environment for training and evaluation. This simulated environment is designed to:

  • Mimic Real-World Scenarios: It includes various dynamic obstacle behaviors (circular, linear, oscillatory, spiral) with changing speeds, radii, and trajectories.
  • Support Interaction: It allows for dynamic interactions between the AUV agent and the environment.
  • Generate Training Data: It dynamically generates training data of varying complexity, crucial for the PPO algorithm.

This custom environment is a key component for validating the framework's performance in challenging, realistic underwater conditions.

Key Insights and Ideas

  • The Power of Hybridization: Combining a physics-inspired planning method (IIFDS) with a powerful learning agent (PPO) is highly effective for complex robotics tasks like AUV trajectory planning. PPO learns how to tune IIFDS for optimal performance.
  • Multi-Objective Rewards are Crucial: Simply rewarding obstacle avoidance and target reaching isn't enough. Including smoothness, dynamics, and energy efficiency in the reward function leads to significantly better, more practical trajectories. Without these, models tend to fall into local optima and produce jerky paths.
  • Explicit Constraints Matter: Directly embedding AUV dynamics and kinematics constraints into the planning process is vital for ensuring that generated trajectories are not just theoretically optimal but physically achievable by the robot.
  • Dynamic Parameter Tuning is Key: Fixed parameters in traditional methods like IIFDS are a bottleneck. PPO's ability to dynamically adjust parameters like repulsion and tangential coefficients allows for much greater adaptability to varying obstacle configurations and environmental dynamics.
  • Simulated Environments Need to be Sophisticated: The paper highlights the importance of creating rich, dynamic simulated environments that accurately reflect real-world challenges to train and validate robust AI systems.

Key Results

The proposed PPO-IIFDS framework demonstrated strong performance across various tests:

  • Robustness in Dynamic Environments: The framework successfully generated smooth, energy-efficient, and collision-free trajectories even in high-density dynamic obstacle scenarios.
  • Superiority over Traditional IIFDS: In static environments, the PPO-IIFDS framework produced more robust and adaptable trajectories compared to the traditional IIFDS with fixed parameters.
  • Impact of Reward Function: Experiments showed that using only obstacle avoidance and target distance rewards resulted in lower learning performance, more fluctuations, and a tendency towards local optima, underscoring the importance of the comprehensive reward function.
  • Efficiency and Generalization: The trained model showed excellent generalization capabilities across different dynamic obstacle combinations and environments.
  • Quantitative Performance: In complex dynamic obstacle environments, the AUV planned collision-free paths with lengths of 15.17m (Environment A) and 13.96m (Environment B), completing planning in 0.17s and 0.14s respectively. This indicates efficient and real-time performance.

Limitations and Future Work

The authors acknowledge several areas for future research:

  • Alternative RL Algorithms: Exploring other advanced continuous deep reinforcement learning algorithms like Soft Actor-Critic (SAC) and TD3 for integration with IIFDS.
  • Hardware Experiments: Conducting real-world underwater experiments to validate the algorithm's feasibility and effectiveness beyond simulation.
  • Advanced AUV Models: Incorporating more complex nonlinear dynamic models and controller characteristics, as the current simplified kinematic model might lead to increased collision risk due to controller delays. Future work could aim for a closed-loop system integrating planning, control, and dynamic modeling.
  • Extension to UAVs: Adapting the PPO-IIFDS framework for Unmanned Aerial Vehicle (UAV) trajectory planning, considering the unique challenges of aerial dynamic environments.

Summary updated

Abstract

Three-dimensional (3D) dynamic trajectory planning for Autonomous Underwater Vehicles (AUVs) is associated with significant challenges such as balancing the trajectory quality, computational efficiency, and environmental adaptability within complex dynamic environments. To tackle these challenges, this paper proposes a novel trajectory planning framework by integrating Proximal Policy Optimization (PPO) and an Improved Interfered Fluid Dynamic System (IIFDS). The IIFDS serves as the planning layer, generating obstacle-adaptive trajectories for AUVs through the dynamic adjustment of flow field parameters. Meanwhile, PPO functions as the learning and decision-making layer, optimizing critical parameters in IIFDS, including repulsion response coefficients, tangential response coefficients, and directional coefficients, to enhance adaptability and real-time decision-making. To meet specific mission requirements, the IIFDS incorporates dynamics and kinematics constraints, while the PPO reward function is improved with a multi-objective dynamic structure. This reward design integrates objectives such as obstacle avoidance, target distance minimization, trajectory smoothness, dynamics constraints, and energy efficiency. These enhancements address sparse reward issues effectively and significantly improve the convergence and practical applicability of trajectory planning. Additionally, a diverse and dynamically complex obstacle environment is constructed for model training and performance evaluation. The experimental results demonstrate that the proposed framework efficiently generates smooth, energy-efficient, and collision-free trajectories in high-density dynamic obstacle scenarios. The framework exhibits strong robustness, excellent generalization capabilities, and offers a reliable solution for 3D dynamic trajectory planning for AUVs.
BibTeX
@article{liu2025three,
  author = {Liqiang Liu and Min Sun and Enjiao Zhao and Kun Zhu},
  title = {{Three-Dimensional Dynamic Trajectory Planning for Autonomous Underwater Robots Under the PPO-IIFDS Framework}},
  journal = {Journal of Marine Science and Engineering},
  year = {2025},
  doi = {10.3390/jmse13030445},
}
Figure