Multi-AUVs Cooperative Target Search Based on Autonomous Cooperative Search Learning Algorithm

Yuan Liu, Min Wang, Zhou Su, Jun Luo, Shaorong Xie, Yan Peng, Huayan Pu, Jiajia Xie, Rui ZhouJournal of Marine Science and Engineering 2020 · DOI

TL;DR

This paper introduces a novel distributed system (DMACSS) and an adaptive learning algorithm (ACSLA) for multi-AUV cooperative target search. By enabling real-time information exchange and employing reinforcement learning, their system significantly improves search accuracy and efficiency compared to existing methods, making AUVs more effective in complex underwater environments.

What Problem Does This Paper Address?

Underwater target search missions, such as maritime rescue or mine countermeasures, are challenging due to the vastness of the search area, the sophistication of the environment, and the limited operational capabilities of individual Autonomous Underwater Vehicles (AUVs). While multi-AUV systems (MAS) offer a promising solution by leveraging distributed and redundant configurations, existing approaches often rely on centralized control (making them vulnerable to single points of failure) and pre-planned search paths that cannot adapt to real-time environmental changes or AUV limitations. This leads to reduced mission efficiency and can be unacceptable for urgent search operations. The paper aims to address these limitations by developing a robust, adaptive, and cooperative search system for multiple AUVs.

Methods

The core of the proposed solution lies in two main components:

  • Distributed Multi-AUVs Collaborative Search System (DMACSS): This system features a distributed control structure, enhancing robustness. It integrates an information fusion mechanism to allow AUVs to share and combine their observations, creating a more comprehensive understanding of the search area. A time-stamp mechanism is also employed to manage information flow effectively, especially under limited communication conditions, and prevent redundant processing of data. The environment is modeled using a probability map, which is dynamically updated based on AUV detections.
  • Autonomous Cooperative Search Learning Algorithm (ACSLA): This is a reinforcement learning (RL)-based algorithm designed to train AUVs to develop optimal cooperative search strategies. ACSLA features a tailored design for:
  • State Information: Combines target probability map information with cooperation information (derived from neighboring AUV positions).
  • Reward Function: A multi-objective reward function that balances accurate target localization with efficient search time, penalizing missed targets and collisions.
  • Training Framework: A novel "Single Asynchronous Sharing Framework" (SASF) is introduced to stabilize the multi-agent RL training process, which is notoriously difficult due to the constantly changing environment perceived by each agent. This framework allows for asynchronous updates and sharing of learned policies.

The paper explores training ACSLA using both Deep Q-Networks (DQN) and Deep Deterministic Policy Gradient (DDPG) algorithms.

Datasets

The paper does not explicitly mention or use pre-existing datasets for training or evaluation. Instead, it relies on a simulated environment built using Python and Google's TensorFlow. This simulation environment is designed to mimic realistic underwater search scenarios, including varying depths and the presence of multiple AUVs.

Key Insights and Ideas

  • Distributed Control is Key for Robustness: Moving away from centralized control in multi-AUV systems is crucial for ensuring system resilience, especially in dynamic and potentially failure-prone underwater environments.
  • Information Fusion and Time-Stamping are Synergistic: Combining local observations through information fusion with a time-stamp mechanism significantly boosts search efficiency by ensuring that AUVs are working with the most relevant and up-to-date information, even with limited communication.
  • Reinforcement Learning for Adaptive Strategies: ACSLA demonstrates that RL can effectively train AUVs to learn complex cooperative search behaviors that adapt in real-time to environmental conditions and the actions of other AUVs, outperforming static or pre-planned methods.
  • Careful Reward Engineering is Essential: The design of the reward function, balancing accuracy and speed, is critical for guiding the RL agent towards desired cooperative search behaviors.
  • Stabilizing Multi-Agent RL is a Major Hurdle: The proposed SASF framework highlights the challenges in training MARL systems and offers a practical solution to improve convergence stability.

Key Results

The simulation tests demonstrated the effectiveness of the DMACSS and ACSLA:

  • DMACSS Stability: The distributed system was shown to run stably.
  • ACSLA Performance: ACSLA consistently outperformed other search methods (random algorithm and coverage control algorithm) in terms of search accuracy and efficiency.
  • Search Accuracy: Achieved an average search accuracy of 99.77%.
  • Convergence Speed: Demonstrated a significantly faster convergence speed, with an average of 145 steps compared to hundreds of steps for other methods.
  • DDPG vs. DQN: ACSLA trained with DDPG showed faster convergence of the probability map compared to DQN, attributed to DDPG's actor-critic architecture directly outputting actions.
  • Impact of AUV Count: Increasing the number of AUVs from 3 to 4 significantly improved convergence speed. However, further increases beyond 6 AUVs showed diminishing returns and even a decrease in convergence speed, likely due to increased computational complexity.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Dynamic Targets: The current system is designed for static targets. Future work should focus on enabling the system to search for dynamic targets.
  • Sophisticated Environments: The current simulation does not fully account for complex environmental factors like obstacles and significant interference, which can impact performance. Future research should address these challenges.
  • Training Time: The SASF framework, while stabilizing training, can lead to long training times due to underutilization of other AUVs' experiences. Further optimization of the training process is needed.
  • Real-World Deployment: While tested in simulation, the transition to real-world deployment will require addressing practical challenges such as communication reliability, sensor noise, and AUV dynamics.

Summary updated

Abstract

As a new type of marine unmanned intelligent equipment, autonomous underwater vehicle (AUV) has been widely used in the field of ocean observation, maritime rescue, mine countermeasures, intelligence reconnaissance, etc. Especially in the underwater search mission, the technical advantages of AUV are particularly obvious. However, limited operational capability and sophisticated mission environments are also difficulties faced by AUV. To make better use of AUV in the search mission, we establish the DMACSS (distributed multi-AUVs collaborative search system) and propose the ACSLA (autonomous collaborative search learning algorithm) integrated into the DMACSS. Compared with the previous system, DMACSS adopts a distributed control structure to improve the system robustness and combines an information fusion mechanism and a time stamp mechanism, making each AUV in the system able to exchange and fuse information during the mission. ACSLA is an adaptive learning algorithm trained by the RL (Reinforcement learning) method with a tailored design of state information, reward function, and training framework, which can give the system optimal search path in real-time according to the environment. We test DMACSS and ACSLA in the simulation test. The test results demonstrate that the DMACSS runs stably, the search accuracy and efficiency of ACSLA outperform other search methods, thus better realizing the cooperation between AUVs, making the DMACSS find the target more accurately and faster.
BibTeX
@article{liu2020multi,
  author = {Yuan Liu and Min Wang and Zhou Su and Jun Luo and Shaorong Xie and Yan Peng and Huayan Pu and Jiajia Xie and Rui Zhou},
  title = {{Multi-AUVs Cooperative Target Search Based on Autonomous Cooperative Search Learning Algorithm}},
  journal = {Journal of Marine Science and Engineering},
  year = {2020},
  doi = {10.3390/jmse8110843},
}
Figure