Abstract
Ocean exploration utilizing autonomous underwater vehicles (AUVs) via reinforcement learning (RL) has emerged as a significant research focus. However, underwater tasks have mostly failed due to the observation delay caused by acoustic communication in the Internet of underwater things. In this study, we present an AoI optimized Markov decision process (AoI-MDP) to improve the performance of underwater tasks. Specifically, AoI-MDP models observation delay as signal delay through statistical signal processing, and includes this delay as a new component in the state space. Additionally, we introduce wait time in the action space, and integrate AoI with reward functions to achieve joint optimization of information freshness and decision-making for AUVs leveraging RL for training. Finally, we apply this approach to the multi-AUV data collection task scenario as an example. Simulation results highlight the feasibility of AoI-MDP, which effectively minimizes AoI while showcasing superior performance in the task.
Introduction
Underwater exploration relies on AUVs communicating via acoustic signals. Unlike radio waves in the air, sound travels slowly underwater, creating significant observation delays. Standard Reinforcement Learning (RL) models usually assume a "zero-wait" strategy: as soon as a robot gets information, it acts. However, in the high-variability environment of the ocean, acting on stale data leads to poor decision-making and "non-causality," where the control policy doesn't align with the actual current state of the world. The authors address the need for Information Freshness, measured by the Age of Information (AoI)—the time elapsed since the last useful piece of data was generated.
Methodology: The AoI-MDP Framework
The authors propose the AoI-MDP, which modifies the standard Markov Decision Process in three critical ways:
- Delay-Aware State Space: Instead of just looking at environmental data, the AUV’s state now includes the specific observation delay ($Y_i$). This allows the RL agent to "know" how old its information is.
- Wait-Time in Action Space: The AUV doesn't just decide where to go; it decides how long to wait ($Z_i$) before the next update. Counter-intuitively, introducing a calculated wait time can actually lower the average AoI in high-variability scenarios by preventing channel congestion.
- AoI-Integrated Rewards: The reward function is penalized by the time-averaged AoI. The agent is trained to maximize task performance while keeping information as fresh as possible.
To make the simulation realistic, they used Statistical Signal Processing (SSP). Instead of picking a random number for delay, they modeled acoustic signal propagation using flow correlators and spatial period graphs to estimate distance and azimuth, mimicking how real sonar works.
Key Results and Insights
The researchers compared AoI-MDP against a standard MDP and tested it across various RL algorithms (SAC for online, CQL for offline).
- Information Freshness: AoI-MDP achieved a significantly lower average AoI (~1.97) compared to standard models.
- Efficiency: The AoI-MDP approach doubled the Sum Data Rate (collecting ~12 units of data vs ~6 for the standard model) while simultaneously reducing energy consumption.
- Reward Optimization: Quantitative results show that AoI-MDP leads to significantly higher cumulative rewards compared to the standard MDP, demonstrating the effectiveness of integrating delay and wait-time into the decision-making process.
- Delay as a Feature: By including the delay ($Y_i$) directly in the state space, the AUV learns to compensate for its own "lag," essentially predicting where targets might have moved during signal travel.
Conclusion
The AoI-MDP framework effectively minimizes AoI while showcasing superior performance in underwater data collection tasks. While the results are promising, the study is primarily conducted in a simulated environment. Future work could involve deploying this framework on physical AUVs in sea trials to validate if the "wait-time" strategy holds up against unpredictable ocean currents and complex multi-path interference.
Summary updated