Research on LSTM-PPO Obstacle Avoidance Algorithm and Training Environment for Unmanned Surface Vehicles

Wan Luo, Xiang Wang, Fang Han, Zhiguo Zhou, Haijun Yu, Lin Zeng, Hong Chen, Jiawei Chen, Xuehua ZhouJournal of Marine Science and Engineering 2025 · DOI

TL;DR

This paper introduces an LSTM-PPO algorithm for Unmanned Surface Vehicle (USV) obstacle avoidance, designed to overcome the limitations of traditional methods in complex, non-ideal marine environments. By integrating Long Short-Term Memory (LSTM) with Proximal Policy Optimization (PPO) and developing a realistic simulation environment, the proposed approach significantly improves navigation efficiency, path planning, and success rates in obstacle avoidance tasks.

What Problem Does This Paper Address?

Current deep reinforcement learning (DRL) algorithms for USV obstacle avoidance often rely on simplified "mass point" models trained in idealized environments. This approach fails to account for crucial real-world factors like ship dynamics and dynamic water conditions (e.g., waves, currents), leading to poor performance and a mismatch between training and actual navigation scenarios. The authors aim to bridge this gap by developing a more robust algorithm and a realistic training environment that better reflects the complexities of the marine setting.

Methods

The core of this research is the LSTM-PPO algorithm, which combines the strengths of two key components:

  • Long Short-Term Memory (LSTM): This recurrent neural network architecture is integrated to process time-series data, allowing the USV to retain and leverage historical environmental information. This is crucial for adapting to dynamic and partially observable environments, improving decision coherence, and mitigating the "short-sightedness" of purely state-based methods.
  • Proximal Policy Optimization (PPO): This policy-gradient algorithm is used for its stable and efficient update mechanism, particularly in continuous action spaces. The paper utilizes the PPO-Clip variant, which helps to prevent drastic policy changes during training, leading to more robust behavior.

The authors also developed a sophisticated deep reinforcement learning training environment in Unity3D. This environment features:

  • A realistic USV physical model incorporating hydrodynamic effects like wave resistance and drift.
  • A virtual LIDAR sensor for detecting static and dynamic obstacles.
  • A multi-level reward function that balances target navigation, obstacle avoidance, and hydrodynamic influences, incorporating a curiosity-driven mechanism.

Datasets

The paper does not explicitly mention pre-existing datasets. Instead, it focuses on creating a novel, custom-built simulation environment within Unity3D. This environment serves as both the training ground and the benchmark for evaluating the proposed algorithm. The environment includes various obstacle distributions and complexities to test the algorithm's adaptability.

Key Insights and Ideas

  • LSTM is crucial for marine navigation: The ability of LSTM to remember past states is not just a nice-to-have; it's essential for USVs to make informed decisions in dynamic and unpredictable water environments, where relying solely on current sensor readings is insufficient.
  • Realistic simulation pays off: Moving beyond idealized models to incorporate hydrodynamic forces, ship dynamics, and varied obstacle types in the training environment significantly boosts the algorithm's real-world applicability and performance.
  • Curiosity-driven rewards enhance learning: Integrating a curiosity-driven mechanism into the reward function helps the USV explore and learn more effectively, leading to better obstacle avoidance strategies.
  • PPO's stability combined with LSTM's memory is a winning formula: The synergy between PPO's robust policy updates and LSTM's historical context processing leads to more stable and efficient decision-making compared to using either method alone or other DRL algorithms.

Key Results

The LSTM-PPO algorithm demonstrated superior performance across several key metrics when compared to A2C, SAC, and PPO in complex environments:

  • Convergence Time: 21.5% reduction in average convergence time (1.5 h for LSTM-PPO vs. 1.8-2.5 h for others).
  • Path Length: 18.5% reduction in average path length (198.52 m for LSTM-PPO vs. 210.37-256.34 m for others).
  • Obstacle Avoidance Success Rate: Approximately 20% enhancement in success rate (86.7% for LSTM-PPO vs. 53.8%-67.4% for others).

The study also showed that LSTM-PPO exhibited the least fluctuation in reward value during training, indicating higher stability.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Computational Complexity: The LSTM structure increases computational overhead, potentially hindering real-time decision-making on low-power embedded devices. Future work will focus on optimizing LSTM models through techniques like Knowledge Distillation and Pruning, and testing on various embedded platforms.
  • Simplified Environment Factors: The simulation environment, while advanced, still simplifies real-world marine factors like wind and currents. Future work aims to incorporate more complex dynamic models for these factors.
  • Single-Boat Focus: The current study focuses on single-boat obstacle avoidance. Future research will extend the algorithm to multi-task USV applications (cruising, target tracking) and multi-vessel collaborative scenarios using Multi-Agent Reinforcement Learning (MARL).
  • COLREGs Integration: The study did not explicitly incorporate the International Regulations for Preventing Collisions at Sea (COLREGs). Future work will integrate COLREG rules into the reward function to ensure adherence to maritime norms.
  • Real-World Testing: The current validation is limited to simulation. Future work will involve real-water testing to assess performance under actual environmental conditions.

Summary updated

Abstract

The current unmanned surface vehicle (USV) intelligent obstacle avoidance algorithm based on deep reinforcement learning usually adopts the mass point model to train in an ideal environment. However, in actual navigation, due to the influence of the ship model and the water surface environment, the training set is triggered. The reward function does not match the actual situation, resulting in a poor obstacle avoidance effect. In response to the above problems, this paper proposes a long and short memory network-proximal strategy optimization (LSTM-PPO) intelligent obstacle avoidance algorithm for non-particle models in non-ideal environments, and designs a corresponding deep reinforcement learning training environment. We integrate the motion characteristics of the unmanned boat and the influencing factors of the surface environment, based on the curiosity-driven set reward function, to improve its autonomous obstacle avoidance ability, combined with the LSTM network to identify and save obstacle information to improve the adaptability to the unknown environment; virtual simulation is performed in Unity. The engine builds a USV physical model and a refined water deep reinforcement learning training environment including a variety of obstacle models. The experimental results demonstrate that the LSTM-PPO algorithm exhibits an effective and rational obstacle avoidance effect, with a success rate of 86.7%, an average path length of 198.52 m, and a convergence time of 1.5 h. A comparison with the performance of three other deep reinforcement learning algorithms reveals that the LSTM-PPO algorithm exhibits a 21.5% reduction in average convergence time, an 18.5% reduction in average path length, and an approximately 20% enhancement in the success rate of obstacle avoidance in complex environments. These results indicate that the LSTM-PPO algorithm can effectively enhance the search efficiency and optimize the path planning in obstacle avoidance for unmanned boats, rendering it more rational.
BibTeX
@article{luo2025research,
  author = {Wan Luo and Xiang Wang and Fang Han and Zhiguo Zhou and Haijun Yu and Lin Zeng and Hong Chen and Jiawei Chen and Xuehua Zhou},
  title = {{Research on LSTM-PPO Obstacle Avoidance Algorithm and Training Environment for Unmanned Surface Vehicles}},
  journal = {Journal of Marine Science and Engineering},
  year = {2025},
  doi = {10.3390/jmse13030479},
}
Figure