Research on Path-Following Technology of a Single-Outboard-Motor Unmanned Surface Vehicle Based on Deep Reinforcement Learning and Model Predictive Control Algorithm

Bin Cui, Yuanming Chen, Xiaobin Hong, Hao Luo, Guanqiao ChenJournal of Marine Science and Engineering 2024 · DOI

TL;DR

This paper introduces a novel path-following control system for single-outboard-motor Unmanned Surface Vehicles (USVs) by combining Deep Deterministic Policy Gradient (DDPG) with Model Predictive Control (MPC). This DDPG-MPC approach significantly improves path-tracking accuracy and stability compared to traditional methods, even in challenging real-world conditions.

What Problem Does This Paper Address?

Autonomous navigation for Unmanned Surface Vehicles (USVs) heavily relies on accurate path-following capabilities. While various methods exist, traditional approaches like Line of Sight (LOS) and Vector Field (VF) have limitations, such as difficulty balancing response time with steady-state error, or issues with rudder oscillation. Furthermore, many advanced control strategies are developed for multi-propeller USVs and are not directly transferable to single-outboard-motor configurations, which have distinct propulsion dynamics and underactuated characteristics. This paper aims to bridge this gap by developing a robust and effective path-following control method specifically for single-outboard-motor USVs, addressing their unique challenges in complex marine environments.

Methods

The core of this research is a hybrid control strategy that integrates Deep Reinforcement Learning (DRL) with Model Predictive Control (MPC).

  • USV and Outboard Motor Modeling: The paper begins by developing detailed kinematic and dynamic models for the single-outboard-motor USV, including its propulsion system. This is crucial for understanding the vehicle's behavior and designing an effective controller.
  • Deep Deterministic Policy Gradient (DDPG) for Guidance: A DDPG algorithm is employed to learn an optimal path-following guidance law. This DRL approach defines a Markov Decision Process (MDP) with a state space that includes longitudinal velocity, heading deviation, cross-track error, and their rates of change, along with the previous action. The action space is the desired change in heading angle. A carefully designed reward function incentivizes minimizing both cross-track and heading errors, promoting stable and accurate path following.
  • Model Predictive Control (MPC) for Heading Control: The desired heading angle generated by the DDPG guidance law is then fed into an MPC-based heading controller. This controller uses a linearized state-space model of the USV and optimizes control inputs (thrust angle) over a prediction horizon to achieve the desired heading while considering system constraints (thrust angle and rate limits). This MPC component ensures smooth and precise execution of the guidance commands.

The combination of DDPG for intelligent, adaptive guidance and MPC for precise, constrained control aims to overcome the limitations of individual methods.

Datasets

The paper does not explicitly mention or introduce new datasets for training or evaluation. Instead, it relies on:

  • Simulation Environment: A MATLAB simulation environment is used to model the USV's dynamics and test the proposed DDPG-MPC algorithm against a traditional ALOS-PID controller.
  • Real Ship Experiments: The algorithm is validated on an independently developed amphibious USV. The experiments were conducted in a harbor environment with varying sea conditions (wind, waves, currents).

Key Insights and Ideas

  • Synergy of DDPG and MPC: The paper demonstrates that combining DDPG for adaptive guidance with MPC for precise control yields superior path-following performance compared to traditional methods. DDPG learns complex behaviors, while MPC handles constraints and optimizes control actions.
  • Importance of State Representation: Including longitudinal velocity and rates of change in the DDPG state space is crucial for capturing the USV's dynamics and enabling more accurate path following, especially in varying conditions.
  • Robustness to Disturbances: The DDPG-MPC approach shows remarkable resilience to environmental disturbances like wind, waves, and currents, a critical factor for real-world USV operations.
  • Improved Longitudinal Velocity Control: The proposed method not only improves path accuracy but also leads to more stable and consistent longitudinal velocity, which is vital for efficient mission execution.

Key Results

The DDPG-MPC algorithm consistently outperformed the traditional ALOS-PID algorithm in both simulation and real-ship experiments:

  • Simulation (Straight Path):
    • Average cross error reduced by approximately 37%.
  • Average heading angle error reduced by approximately 21%.
  • Simulation (Multi-polyline Path):
    • Average cross error was about 8.5% smaller.
  • Average heading angle error was about 4.6% higher (attributed to a smoother, less oscillatory approach).
  • Real Ship Experiments:
    • Maximum cross error was approximately one-third of the ALOS-PID.
  • Average cross error was 6.7% lower (6.747 m vs. 7.233 m).
  • Average heading angle error was 60.0% lower (13.8° vs. 34.6°).
  • The DDPG-MPC controlled USV maintained a higher and more stable longitudinal velocity over extended periods compared to the ALOS-PID.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the DDPG-MPC algorithm shows a slightly larger crossover error at path turning points in multi-polyline paths, this difference is considered negligible in the overall context and is a trade-off for a smoother, less oscillatory trajectory.
  • The study focuses on a single-outboard-motor USV; its direct applicability to other USV configurations might require further adaptation.
  • Future Work:
    • The authors plan to further optimize the algorithm to enhance its real-time performance and robustness.
  • The goal is to make the system applicable to more complex real-world navigation scenarios, aiming for safer, more efficient, and stable path-following solutions.

Summary updated

Abstract

Path following is one of the key technologies for unmanned surface vehicles (USVs). This paper proposes a path-tracking control method for a single-outboard-motor USV based on a Deep Deterministic Policy Gradient (DDPG) algorithm and model predictive control (MPC) algorithm. Initially, the motion model and outboard motor model of the USV are analyzed. Subsequently, simulation and real ship experiments provide a comprehensive performance comparison between the proposed DDPG-MPC method and the traditional ALOS-PID method. The results indicate that for straight path tracking, the DDPG-MPC algorithm achieves 37% and 21% reductions in the average cross error and heading angle error, respectively, compared to the ALOS-PID algorithm. The real ship experiments further validate the DDPG-MPC algorithm’s advantages in real-world environments. Specifically, under disturbances like wind, waves, and currents, the maximum cross error of the DDPG-MPC algorithm is one-third of the ALOS-PID algorithm. Additionally, the DDPG-MPC algorithm sustains a higher and more stable longitudinal velocity over extended periods, while the ALOS-PID algorithm shows greater instability and variability. Overall, the findings confirm the feasibility and effectiveness of the proposed approach, highlighting its potential for enhancing path-tracking control performance in single-outboard-motor USVs.
BibTeX
@article{cui2024research,
  author = {Bin Cui and Yuanming Chen and Xiaobin Hong and Hao Luo and Guanqiao Chen},
  title = {{Research on Path-Following Technology of a Single-Outboard-Motor Unmanned Surface Vehicle Based on Deep Reinforcement Learning and Model Predictive Control Algorithm}},
  journal = {Journal of Marine Science and Engineering},
  year = {2024},
  doi = {10.3390/jmse12122321},
}
Figure