Deep Reinforcement Learning–Based Online One-to-Multiple Charging Scheme in Wireless Rechargeable Sensor Network

Zheng Gong, Hao Wu, Yong Feng, Nianbo LiuSensors 2023 · DOI

TL;DR

This paper introduces OTM3DQN, a novel deep reinforcement learning approach for optimizing charging schedules in wireless rechargeable sensor networks (WRSNs). By enabling one-to-multiple charging and intelligently determining both the charging sequence and the amount of energy transferred, OTM3DQN significantly reduces node deaths and charging latency, making it more scalable for large WRSNs.

What Problem Does This Paper Address?

Wireless Sensor Networks (WSNs) are crucial for various applications, but their limited battery life is a major bottleneck. Wireless Rechargeable Sensor Networks (WRSNs) offer a solution by using mobile chargers (MCs) to replenish energy. However, existing charging schemes often suffer from:

  • One-to-one charging limitations: MCs can only charge one node at a time, leading to long waiting times and node failures, especially in large networks.
  • Suboptimal scheduling: Many schemes don't holistically optimize the MC's path and charging decisions, leading to inefficiencies.
  • Lack of charging amount optimization: Existing methods often provide a fixed amount of charge, which can be too much (increasing charging time) or too little (not sustaining node operation), impacting overall network performance.
  • Inflexibility in dynamic environments: Pre-defined charging paths struggle to adapt to the unpredictable energy consumption patterns of sensor nodes.

This paper aims to address these issues by proposing an online, one-to-multiple charging scheme that dynamically optimizes both the charging sequence and the amount of energy delivered.

Methods

The core of the proposed solution is an Online One-to-Multiple (OTM) charging scheme powered by Deep Reinforcement Learning (DRL), specifically a Double Dueling Deep Q-Network (3DQN). Here's a breakdown:

  • Network Cellularization: The network is divided into hexagonal "charging cells" based on the MC's effective charging range. This simplifies the decision-making process by treating cells as discrete units.
  • 3DQN for Scheduling: The 3DQN algorithm is employed to learn an optimal policy for selecting the next charging cell. It combines:
  • Deep Q-Network (DQN): Uses neural networks to approximate Q-values, handling high-dimensional state spaces.
  • Double DQN: Mitigates the overestimation bias inherent in standard DQN by decoupling action selection and value estimation.
  • Dueling DQN: Improves learning stability by separating the estimation of state value and action advantage.
  • Joint Optimization: Unlike previous works, OTM3DQN jointly optimizes:
  • Charging Sequence: Decides the order in which charging cells are visited.
  • Charging Amount: Dynamically determines how much energy to deliver to nodes within a cell, considering their energy demand, network survival time, and the MC's residual energy.
  • Prioritized Experience Replay: Used to improve training efficiency by sampling more informative transitions from the experience buffer.

Datasets

Not discussed in the paper. The experiments were conducted using a Python simulator with custom-generated network environments of varying sizes (50, 100, and 200 nodes).

Key Insights and Ideas

  • One-to-Multiple Charging is Key for Scalability: The paper strongly advocates for one-to-multiple charging as a fundamental shift needed to handle the energy demands of large-scale WRSNs, moving beyond the limitations of one-to-one approaches.
  • Holistic Optimization is Crucial: Simply optimizing the charging path or sequence isn't enough. The true innovation lies in jointly optimizing both the sequence and the amount of energy delivered, leading to more efficient and effective charging.
  • DRL as a Natural Fit for Dynamic Environments: The paper highlights how DRL, particularly with advanced architectures like 3DQN, is well-suited to adapt to the dynamic and high-dimensional nature of WRSN environments, outperforming traditional methods.
  • Balancing Energy Delivery: The dynamic adjustment of charging amount is critical. It's not just about giving nodes energy, but giving them the right amount to ensure they don't die before the next charge, while also not wasting MC time on overcharging.

Key Results

The OTM3DQN scheme demonstrated superior performance compared to several existing typical works (FCFS, Greedy, RMP-RL, ACRL) across various metrics:

  • Reduced Node Dead Ratio: OTM3DQN significantly lowered the percentage of dead nodes, especially in larger networks.
  • Lower Charging Latency: The time from a node requesting charge to receiving it was substantially reduced.
  • Improved Energy Efficiency: The scheme achieved higher energy efficiency, meaning more of the MC's energy was effectively transferred to the sensor nodes.
  • Faster and More Stable Training: The use of Double DQN and Dueling DQN contributed to a faster convergence rate and more stable training process for the DRL agent.
  • Scalability: The proposed scheme showed better performance as the network size increased, indicating its scalability for large-scale WRSNs. For instance, in a 200-node network, OTM3DQN achieved an average cost of 132.79m and an energy efficiency of 49.57%, outperforming other methods.

Limitations and Future Work

  • Limitations:
    • The paper focuses on a single MC. The complexity of coordinating multiple MCs is not addressed.
  • The current scheme is non-preemptive, meaning an MC completes a charging task before starting a new one.
  • Future Work:
    • Extend the scheme to handle multiple MCs for even larger and more complex WRSNs.
  • Apply the algorithm to real-world WRSN deployments to validate its effectiveness in practical scenarios.

Summary updated

Abstract

Wireless rechargeable sensor networks (WRSN) have been emerging as an effective solution to the energy constraint problem of wireless sensor networks (WSN). However, most of the existing charging schemes use Mobile Charging (MC) to charge nodes one-to-one and do not optimize MC scheduling from a more comprehensive perspective, leading to difficulties in meeting the huge energy demand of large-scale WSNs; therefore, one-to-multiple charging which can charge multiple nodes simultaneously may be a more reasonable choice. To achieve timely and efficient energy replenishment for large-scale WSN, we propose an online one-to-multiple charging scheme based on Deep Reinforcement Learning, which utilizes Double Dueling DQN (3DQN) to jointly optimize the scheduling of both the charging sequence of MC and the charging amount of nodes. The scheme cellularizes the whole network based on the effective charging distance of MC and uses 3DQN to determine the optimal charging cell sequence with the objective of minimizing dead nodes and adjusting the charging amount of each cell being recharged according to the nodes' energy demand in the cell, the network survival time, and MC's residual energy. To obtain better performance and timeliness to adapt to the varying environments, our scheme further utilizes Dueling DQN to improve the stability of training and uses Double DQN to reduce overestimation. Extensive simulation experiments show that our proposed scheme achieves better charging performance compared with several existing typical works, and it has significant advantages in terms of reducing node dead ratio and charging latency.
BibTeX
@inproceedings{gong2023deep,
  author = {Zheng Gong and Hao Wu and Yong Feng and Nianbo Liu},
  title = {{Deep Reinforcement Learning–Based Online One-to-Multiple Charging Scheme in Wireless Rechargeable Sensor Network}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23083903},
}
Figure