A Novel Sensor Scheduling Algorithm Based on Deep Reinforcement Learning for Bearing-Only Target Tracking in UWSNs

Linyao Zheng, Meiqin Liu, Senlin Zhang, Jian LanIEEE caa J. Autom. Sinica 2023 · DOI

Introduction

Underwater Wireless Sensor Networks (UWSNs) are critical for ocean exploration and monitoring. A primary application is target tracking, which often relies on bearing-only measurements due to the complexities of the underwater environment. However, the limited battery capacity of underwater sensors necessitates efficient scheduling strategies to balance tracking accuracy with energy consumption. Traditional methods like Dynamic Programming (DP) or Genetic Algorithms (GA) often require precise environmental models, which are difficult to obtain in dynamic underwater settings.

System Model and Problem Formulation

The tracking scenario involves a target moving through a 3D underwater volume monitored by a cluster-based sensor network. Each cluster consists of a Cluster Head (CH) and multiple Cluster Members (CMs). The CMs act as bearing-only sensors, providing angular information about the target's position to the CH.

System model of bearing-only target tracking in a cluster-based UWSN.

The scheduling problem is formulated as a Markov Decision Process (MDP), where the objective is to select an optimal subset of sensors at each time step to minimize tracking error while maximizing network lifetime.

Proposed D3QN-PER Algorithm

To address the limitations of model-based approaches, this paper proposes a Dueling Double Deep Q-Network (D3QN) architecture enhanced with Prioritized Experience Replay (PER).

D3QN and PER Architecture

The D3QN architecture decouples the estimation of state value and action advantage, allowing the agent to learn which states are inherently valuable regardless of the action taken. The "Double" Q-learning component mitigates the overestimation bias common in standard Q-learning, leading to more stable training. Furthermore, PER prioritizes transitions with higher temporal-difference (TD) errors, ensuring the agent learns more frequently from informative experiences.

Reward Function and Mock Data Method

A significant challenge in real-time tracking is the lack of "ground truth" for reward calculation. The authors introduce a "mock data" method, which evaluates tracking quality by comparing estimated states against actual sensor measurements. The reward function specifically weights Fisher Information (a measure of tracking quality) against energy consumption, allowing for a tunable balance between accuracy and battery life.

Results and Discussion

The performance of the D3QN-PER algorithm was evaluated in a simulated 1000m x 1000m x 1000m underwater environment and compared against DP and GA baselines.

Comparison of tracking accuracy (RMSE) and energy consumption across different scheduling algorithms.

As shown in the results, the D3QN-PER approach achieves a lower Root Mean Square Error (RMSE) compared to traditional methods. Furthermore, it demonstrates superior energy efficiency, maintaining high tracking performance with lower cumulative energy expenditure. The agent successfully converged to a stable, high-reward policy after approximately 130 training episodes.

Conclusion

The proposed D3QN-PER algorithm provides a robust, model-free solution for sensor scheduling in UWSNs. By effectively balancing tracking precision and energy conservation, it outperforms conventional optimization techniques without requiring perfect prior knowledge of the environment. Future work will focus on adapting the algorithm to maneuvering targets and more complex environmental noise conditions.

Citation: L. Y. Zheng, M. Q. Liu, S. L. Zhang, and J. Lan, “A novel sensor scheduling algorithm based on deep reinforcement learning for bearing-only target tracking in UWSNs,” IEEE/CAA J. Autom. Sinica, vol. 10, no. 4, pp. 1077–1079, Apr. 2023.

Summary updated

BibTeX
@article{zheng2023novel,
  author = {Linyao Zheng and Meiqin Liu and Senlin Zhang and Jian Lan},
  title = {{A Novel Sensor Scheduling Algorithm Based on Deep Reinforcement Learning for Bearing-Only Target Tracking in UWSNs}},
  journal = {IEEE caa J. Autom. Sinica},
  year = {2023},
  doi = {10.1109/jas.2023.123159},
}
Figure