An Offline Reinforcement Learning Approach for Path Following of an Unmanned Surface Vehicle

Zexing Zhou, Tao Bao, Jun Ding, Yihong Chen, Zhengyi Jiang, Bo ZhangJournal of Marine Science and Engineering 2024 · DOI

TL;DR

This paper introduces SAC-N, a novel offline reinforcement learning (RL) approach for Unmanned Surface Vehicle (USV) path following. By leveraging pre-collected datasets and a diversified Q-ensemble within the Soft Actor-Critic (SAC) framework, SAC-N achieves robust and efficient path following without requiring online interaction, offering a safer and faster training alternative to existing methods.

What Problem Does This Paper Address?

Autonomous navigation for Unmanned Surface Vehicles (USVs) is increasingly important for tasks like marine monitoring. A critical component of this autonomy is accurate path following, which involves navigating a USV along a predefined trajectory. Traditional control methods often rely on precise mathematical models of the vehicle and its environment, which can be difficult to obtain and may not account for complex real-world disturbances. While online reinforcement learning (RL) offers a way to learn control policies without explicit models, it requires extensive, potentially risky, and costly real-time interaction with the environment. This poses a significant challenge for USVs, where exploratory actions during training could lead to damage or environmental hazards. The paper addresses this by proposing an offline RL approach, which learns from pre-existing data, thereby avoiding the risks and costs associated with online learning.

Methods

The core of this paper is the development of a Soft Actor-Critic with diversified Q-ensemble (SAC-N) steering controller. This approach builds upon the Soft Actor-Critic (SAC) algorithm, known for its ability to balance reward maximization with entropy maximization for improved exploration and stability. The key innovations are:

  • Offline Learning: Instead of learning through real-time interaction, SAC-N is trained on a static, pre-collected dataset of USV path-following trials. This significantly enhances safety and reduces training costs.
  • Diversified Q-Ensemble: To mitigate the "extrapolation error" common in offline RL (where the agent encounters data outside its training distribution), SAC-N employs an ensemble of N Q-networks. It then uses the minimum Q-value from this ensemble to make more conservative and robust policy updates. This pessimistic estimation strategy is crucial for preventing overestimation of Q-values, which can lead to unstable policies.
  • Actor-Critic Architecture: The controller uses a standard actor-critic framework. The actor network learns the policy (steering commands), while the critic networks (the Q-ensemble) evaluate the actions taken by the actor.
  • Markov Decision Process (MDP) Formulation: A specific MDP is defined for the USV path-following task, with states including errors in heading and cross-track position, and actions corresponding to steering commands.

Datasets

The paper utilizes real-world trial datasets collected from the USV-900, an unmanned surface vehicle developed by the China Ship Scientific Research Center. These datasets consist of approximately 200,000 transitions, each containing state, action, reward, and next state information. The data was collected during manual operation of the USV and organized into a dataset for offline training. No new benchmark datasets were introduced; rather, existing real-world operational data was repurposed for offline RL training.

Key Insights and Ideas

  • Offline RL for USVs is Viable and Beneficial: The paper convincingly demonstrates that offline RL can be effectively applied to USV path following, offering a safer and more efficient alternative to online methods.
  • Q-Ensemble is Crucial for Offline Stability: The diversified Q-ensemble is highlighted as a key component for achieving stable and robust performance in offline RL. By using the minimum Q-value, the algorithm avoids the pitfalls of overestimation, which is particularly important when dealing with potentially out-of-distribution data.
  • Balancing Performance and Efficiency: SAC-N achieves performance comparable to established controllers like NMPC and TD3, but with significantly reduced training time and without the need for online interaction. This makes it a highly practical solution for real-world deployment.
  • Steering Command Reduction: The proposed controller generates smaller steering commands compared to some other methods, which translates to reduced axial thrust loss and potentially improved energy efficiency.

Key Results

  • Simulation: In simulations, SAC-N demonstrated comparable or superior performance to NMPC and TD3 in terms of steady-state error, with errors approaching zero after stable line tracking. It also generated smaller steering commands than TD3.
  • Real-World Tests:
    • SAC-N achieved performance comparable to NMPC and TD3 in path-following tests on a full-scale USV.
  • While NMPC offered the highest tracking accuracy, it demanded significant CPU time.
  • SAC-N demonstrated a favorable balance between precise tracking and steering maneuverability, with significantly smaller steering command angles than NMPC, leading to reduced axial thrust loss.
  • Training Efficiency: SAC-N was trained offline in just 1.2 hours, whereas TD3 required 7.5 hours of online training for the same number of frames. This represents a substantial saving in training time and cost.
  • Performance with Ensemble Size: Increasing the Q-ensemble size (N) in SAC-N led to improved average returns, with N=50 outperforming N=10 and N=20.

Limitations and Future Work

  • Limitations:
    • While SAC-N's performance is comparable to TD3, TD3 still shows slightly better performance in terms of Mean Absolute Error (MAE) indicators in some comparisons.
  • The paper acknowledges that all controllers struggled to consistently converge to zero error in real-world tests due to random disturbances from wind and wave loads.
  • Future Work:
    • The authors plan to extend the capabilities of their approach to collision avoidance and more complex tasks.
  • They also aim to explore training offline RL algorithms using vision-based inputs, which could further enhance the autonomy of USVs.

Summary updated

Abstract

Path following is crucial for enhancing the autonomy of unmanned surface vehicles (USVs) in water monitoring missions. This paper presents an offline reinforcement learning (RL) controller for USVs. The controller employs the soft actor–critic algorithm with a diversified Q-ensemble to optimize the steering control policy using a pre-collected dataset of USV path-following trials. A Markov decision process (MDP) tailored for path following is formulated. The proposed offline RL steering controller, trained on static datasets, demonstrates improved sample efficiency and asymptotic performance due to an expanded ensemble of Q-networks. The accuracy and adaptive learning capabilities of the RL controller are validated through simulations and free-running tests.
BibTeX
@article{zhou2024offline,
  author = {Zexing Zhou and Tao Bao and Jun Ding and Yihong Chen and Zhengyi Jiang and Bo Zhang},
  title = {{An Offline Reinforcement Learning Approach for Path Following of an Unmanned Surface Vehicle}},
  journal = {Journal of Marine Science and Engineering},
  year = {2024},
  doi = {10.3390/jmse12122173},
}
Figure