TL;DR
This paper presents a novel approach for Unmanned Surface Vehicles (USVs) to dynamically avoid obstacles by leveraging cross-domain deep reinforcement learning (DRL). By first training a DRL agent on a ground robot and then fine-tuning it on a USV, the authors significantly reduce training time and improve performance, demonstrating a practical method for transferring learned behaviors to challenging marine environments.
What Problem Does This Paper Address?
Autonomous navigation for Unmanned Surface Vehicles (USVs) is crucial, but avoiding dynamic obstacles in the marine environment presents unique challenges. The fluid dynamics of water lead to significant damping and inertia, making rapid maneuvers difficult. Furthermore, USVs are often underactuated, limiting their agility. Traditional path planning methods can be computationally expensive and struggle with dynamic environments. While Deep Reinforcement Learning (DRL) shows promise for obstacle avoidance, training DRL agents from scratch in the real world is time-consuming, expensive, and data-scarce due to the difficulty of deploying and operating USVs. This paper addresses the need for an efficient and robust obstacle avoidance system for USVs that can be trained effectively, even in data-poor marine environments, by leveraging knowledge gained from more accessible ground-based systems.
Methods
The core of the proposed solution is a two-tiered control system:
- High-Level Control (DRL Agent): A Deep Q Network (DQN) is employed as the DRL agent. To handle the sequential nature of dynamic obstacle avoidance, a Long Short-Term Memory (LSTM) layer is integrated into the DQN architecture. This allows the agent to process a sequence of past observations (Lidar scans, relative goal position, vehicle velocity) to better predict obstacle motion. The agent's action space is discrete, consisting of five predefined waypoints that guide the vehicle.
- Low-Level Control (NN-MPC): To achieve precise control and reject disturbances (like wind) not explicitly modeled by the DRL agent, a Neural Network Model Predictive Controller (NN-MPC) is used. The NN-MPC learns a model of the USV's dynamics and uses it to predict future states and generate control commands to track the temporary waypoint provided by the DRL agent. This hybrid approach combines the reactive decision-making of DRL with the robust tracking capabilities of MPC.
A key innovation is the cross-domain training strategy. The DQN is initially trained on a UGV in a simulated ground environment, which is more data-rich and easier to simulate. The learned policy is then transferred to a USV simulation and further trained. This transfer learning approach significantly reduces the overall training time and improves performance compared to training solely in the USV domain.
Datasets
The paper utilizes simulated environments for training and testing:
- UGV Simulation: A 4m x 4m square environment with dynamic obstacles (0.3-0.5m in size, up to 0.5 m/s velocity) was used for initial DRL training.
- USV Simulation: Adapted from the Virtual RobotX (VRX) simulator, this environment features larger obstacles (4m average size) and higher velocities (up to 5 m/s). The Lidar range was increased to 15m for better early detection.
- Real-world UGV: Used for validation, involving navigation between two waypoints with a walking human as a dynamic obstacle.
- Real-world USV: The BREAM USV was used for field tests, navigating a 100m course with a dinghy as a dynamic obstacle.
No new benchmark datasets were introduced; rather, existing simulation environments were adapted and augmented for the specific training and testing needs of the cross-domain approach.
Key Insights and Ideas
- Cross-domain transfer learning is highly effective for USV navigation: Training a DRL agent on a UGV first dramatically reduces the time and data required to achieve good performance on a USV, overcoming the data scarcity issue in marine environments.
- LSTM is crucial for dynamic obstacle avoidance: Integrating LSTMs into the DQN allows the agent to effectively capture the temporal dynamics of moving obstacles, which is essential for predictive avoidance.
- Hybrid DRL-MPC offers robust performance: Combining the high-level decision-making of DRL with the precise tracking and disturbance rejection of NN-MPC creates a powerful system capable of handling complex scenarios, including environmental disturbances like wind.
- Sim-to-real transfer is facilitated: The methodology demonstrates successful transfer from simulation to real-world USV operation, indicating the robustness of the learned policy.
Key Results
- Reduced Training Time: Cross-domain training decreased training time by 28% compared to training solely in the USV domain.
- Improved Performance: The cross-domain approach resulted in a 70-point increase in mean episodic reward.
- High Success Rate: The trained agent successfully avoided dynamic obstacles in both simulated and real-world tests, achieving a mean path deviation of around 1 with respect to the obstacle size.
- Disturbance Rejection: The NN-MPC controller demonstrated effective rejection of wind disturbances, allowing the USV to avoid collisions even when facing challenging wind conditions (up to 6 m/s).
Limitations and Future Work
- Single Obstacle Avoidance: The current DRL agent is designed to avoid one obstacle at a time. The authors acknowledge that it may fail in scenarios with multiple simultaneous moving objects.
- Future Work:
- Train the agent to handle scenarios with multiple moving obstacles.
- Explore more advanced DRL models like Actor-Critic and Proximal Policy Optimization (PPO) for generating continuously augmented waypoints.
- Incorporate higher-dimension sensors such as 3D Lidar and sonar for more complex environmental perception.
- Further investigate the feasibility of transitioning between domains in even more complex settings.
Summary updated