Abstract
We investigate the scenario that a robot needs to reach a designated goal after taking a sequence of appropriate actions in a non-static environment that is partially structured. One application example is to control a marine vehicle to move in the ocean. The ocean environment is dynamic and oftentimes the ocean waves result in strong disturbances that can disturb the vehicle's motion. Modeling such dynamic environment is non-trivial, and integrating such model in the robotic motion control is particularly difficult. Fortunately, the ocean currents usually form some local patterns (e.g. vortex) and thus the environment is partially structured. The historically observed data can be used to train the robot to learn to interact with the ocean tidal disturbances. In this paper we propose a method that applies the deep reinforcement learning framework to learn such partially structured complex disturbances. Our results show that, by training the robot under artificial and real ocean disturbances, the robot is able to successfully act in complex and spatiotemporal environments.
Introduction
Navigating the open ocean presents a significant challenge for low-power autonomous underwater vehicles (AUVs). These robots often operate at speeds comparable to or slower than the currents they encounter. Furthermore, ocean dynamics are spatiotemporal, changing across both space and time in ways that are difficult to capture with standard mathematical models.
Traditional control techniques, such as the iterative Linear Quadratic Regulator (iLQR), rely on precise environmental models. However, when the environment is "unstructured" or highly turbulent, these models often fail. This research explores whether historical data can be leveraged to teach a robot to recognize recurring patterns in the water—such as vortices and meanders—and use them to its advantage.
Methodology
The proposed approach utilizes a Deep Reinforcement Learning (DRL) framework with a specialized dual-subnet neural network architecture designed to handle complex disturbances.
Dual-Subnet Architecture
The network is divided into two primary components:
- Environmental Feature Extraction: This subnet uses Recurrent Neural Network (RNN) layers to extract spatiotemporal features from the environment, effectively identifying the current state of the water.
- Decision-Making: This subnet uses Fully Connected (FC) layers to process the extracted features alongside the vehicle's state to determine the optimal action.
Input and Training
The model processes a three-channel map representing local current velocities (x and y axes) and goal/obstacle locations. It also incorporates the vehicle’s velocity and orientation. Training is conducted using a policy gradient method, enhanced by Importance Sampling for faster convergence and Experience Replay to maintain stability and prevent overfitting.
Experimental Results
The DRL agent was evaluated against a baseline iLQR controller using both synthetic and real-world data.
- Artificial Disturbances: In synthetic vector fields representing vortices, meanders, and centripetal patterns, the DRL approach achieved a 96% success rate, compared to 92% for iLQR.
- Real-World Ocean Data: Using data from the Regional Ocean Model System (ROMS) for the Los Angeles coast, the DRL agent demonstrated superior efficiency, reducing travel time by approximately 20% and total steps by 10%.
- Reliability: Unlike the iLQR controller, which showed high performance variance, the DRL agent remained consistently effective across various starting and goal positions.
Conclusion and Future Work
This study demonstrates that DRL can effectively exploit the "partial structure" of ocean currents to improve AUV navigation. By learning to "surf" recognizable flow patterns, robots can reach goals more reliably than with traditional model-based methods. Future research will focus on extending these models to 3D environments and improving robustness against unobserved weather events.
Summary updated