Underwater environments present a unique set of challenges for robotic control. For the "Aqua" hexapod—a six-legged swimming robot—navigating these waters requires managing complex, interconnected dynamics. Traditional control methods often struggle with the Multi-Input Multi-Output (MIMO) nature of such systems, where a single movement can trigger unintended secondary effects. This paper explores a centralized Deep Q-Network (DQN) controller designed to replace traditional decoupled PID controllers for underwater object tracking, demonstrating significant improvements in robustness and efficiency.
The Challenge: MIMO Dynamics and Decoupled Control
Controlling a 6-DOF robot underwater is inherently difficult due to the coupling of movements. In the case of the Aqua robot, an action intended to change yaw (turning left or right) frequently induces a change in pitch (tilting up or down). Standard engineering practice relies on Proportional-Integral-Derivative (PID) controllers. However, these are typically "decoupled," meaning separate PID loops manage yaw and pitch independently without communication.
In high-latency underwater environments, these independent loops often work at cross-purposes, leading to instability. Furthermore, PID controllers are rigid; they require manual retuning if the robot’s physical state changes (e.g., a damaged actuator) or if environmental conditions like water density fluctuate. The researchers sought to develop a "centralized brain" capable of learning these interdependencies and adapting to changes dynamically.
Hybrid Control Architecture
The proposed system utilizes a hybrid architecture that separates the tracking task into vision and control modules, combining the strengths of deep learning with classical control.
1. Vision and Tracking
The vision module employs YOLOv7 for real-time object detection (specifically targeting divers) and SORT (Simple Online and Realtime Tracking) to maintain a temporal lock on the target. This process reduces high-dimensional video input into low-dimensional bounding box coordinates, which serve as the primary input for the controller.
2. Centralized DQN Controller
The core of the system is a Deep Q-Network (DQN) that manages angular rates (yaw and pitch). Unlike the decoupled PID approach, the DQN features two "heads" that share internal parameters, allowing the agent to learn how yaw and pitch actions influence one another.
A critical component of this controller is the history buffer. Because underwater systems suffer from significant mechanical and fluid-dynamic lag, the researchers included a history of the previous 20 frames in the state input. This restores the "Markov property," providing the agent with the context necessary to understand how past actions affect the current state.
3. Safe Exploration and Hybrid Execution
To mitigate the risks associated with the early stages of reinforcement learning training, the system used a classical PID controller as a safety net. Control was gradually transitioned from the PID to the DQN as the agent's performance improved. In the final deployment, the RL agent handles orientation, while a standard PID handles linear velocity.
Experimental Results
The system was evaluated within AquaSim, a Unity-based ROS2 simulator that models complex hydrodynamics, including thrust, buoyancy, and linear drag.
- Tracking Performance: The DQN-based controller demonstrated superior endurance. In long-term tracking trials, the RL agent maintained a lock for 45,151 frames, significantly outperforming the 19,361 frames achieved by the PID controller.
- Reward and Efficiency: The cumulative reward—a metric representing how effectively the target remained centered in the frame—was nearly twice as high for the DQN compared to the PID in scenarios involving random target movement.
- Fault Tolerance: One of the most significant findings was the agent's inherent robustness. When researchers simulated a "broken leg" (actuator failure), the RL agent naturally compensated for the loss of symmetry. Because the agent was trained on the system's actual response rather than a static mathematical model, it maintained stability where the PID controller's performance tanked.
- Environmental Adaptability: The DQN successfully managed 20% variations in environmental parameters, such as buoyancy and damping, without requiring the manual recalibration that a PID system would necessitate.
Conclusion and Future Directions
The transition from decoupled PID loops to a centralized RL-based controller marks a significant step forward in underwater robotics. By learning the complex interactions of a 6-DOF system, the DQN provides a level of robustness and tracking precision that classical methods struggle to match.
However, limitations remain. The current DQN implementation uses discrete actions, which can lead to less fluid movement compared to continuous control. Future research will explore algorithms like Proximal Policy Optimization (PPO) or Soft Actor-Critic (SAC) to address this. Additionally, while the simulation results are compelling, the "sim-to-real" gap remains a hurdle; future work will focus on testing the system against the unpredictable turbulence of open-water environments.
Summary updated