Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-aided and Wireless-powered Wireless Networks

Shimin Gong, Leiyang Cui, Bo Gu, Bin Lyu, Dinh Thai Hoang, Dusit Niyato

Abstract

This paper investigates the minimization of Age-of-Information (AoI) in wireless-powered sensor networks (WPSNs) enhanced by Intelligent Reflecting Surfaces (IRS). We propose a hierarchical deep reinforcement learning (DRL) framework that decouples discrete scheduling decisions from continuous beamforming optimization. By employing Proximal Policy Optimization (PPO) in the outer loop for scheduling and efficient optimization techniques in the inner loop for beamforming, the framework achieves significantly fresher data updates. Numerical results show a 53.8% reduction in average AoI compared to baseline methods, highlighting the effectiveness of the hierarchical approach in managing the complex interplay between energy harvesting and data transmission.

Introduction

In modern Internet of Things (IoT) applications, the timeliness of data, measured by Age-of-Information (AoI), is often more critical than traditional metrics like throughput. AoI represents the time elapsed since the last successful update was generated at a source. Minimizing AoI is particularly challenging in Wireless-Powered Sensor Networks (WPSNs), where nodes lack batteries and must harvest energy from an Access Point (AP) before transmitting data.

The integration of Intelligent Reflecting Surfaces (IRS) offers a way to enhance both energy transfer and data transmission by intelligently shaping the wireless environment. However, this introduces a complex optimization problem involving discrete scheduling (which node transmits?), mode selection (downlink power vs. uplink data), and high-dimensional continuous beamforming (AP and IRS configurations). Standard DRL approaches often struggle to converge in such mixed-integer, high-dimensional action spaces.

Methodology

The proposed solution utilizes a Hierarchical DRL framework to "divide and conquer" the optimization problem.

System Model and Notations

The system consists of an AP, an IRS, and multiple IoT devices. The key variables and notations used in the formulation are summarized below:

Table of notations used in the system model and optimization problem.

Hierarchical Framework

  1. Outer-Loop (Scheduling): A DRL agent (e.g., PPO or DQN) manages discrete decisions. It selects the AP's mode (downlink energy harvesting or uplink data transmission) and schedules specific sensor nodes for transmission.
  2. Inner-Loop (Optimization): Given the scheduling decisions from the outer loop, the inner loop optimizes the continuous variables—specifically the AP beamforming vectors and IRS phase shifts. This is achieved using Alternating Optimization (AO) and Semi-Definite Relaxation (SDR).
  3. Simple Approximation: To reduce the computational overhead of solving complex optimizations during training, a "Simple Approximation" method was developed. This approach uses channel gain ordering and lower-bound approximations to maintain performance while significantly speeding up the training process.

Simulation Setup

The framework was evaluated using numerical simulations in a 3D environment. Sensor nodes ($K=3$ to $15$) were randomly distributed, and the IRS size ($N$) varied from 20 to 100 elements. A Rayleigh fading model with path-loss parameters was employed to simulate realistic channel conditions.

Key Results and Discussion

The hierarchical approach demonstrates several advantages over conventional methods:

  • Stability and Convergence: Decoupling scheduling logic from physical layer optimization allows the PPO agent to converge more effectively than "all-in-one" agents that attempt to learn both simultaneously.
  • Significant AoI Reduction: The hierarchical PPO method achieved a 53.8% reduction in average AoI compared to energy-priority baselines. It also outperformed Round-Robin (ROR) scheduling by better adapting to dynamic data arrivals and energy states.
  • Impact of IRS: Increasing the number of IRS elements ($N$) directly reduces transmission failure rates. As the channel quality improves with larger IRS arrays, the system bottleneck shifts from physical transmission constraints to scheduling delays.
  • Energy-AoI Trade-off: The system allows for tuning a trade-off parameter ($\lambda_0$), enabling a balance between aggressive data updates (low AoI) and conservative energy management to prevent node depletion.

Conclusion

This work demonstrates that hierarchical DRL is a powerful tool for managing the complexities of IRS-aided WPSNs. By separating scheduling from beamforming, the framework ensures data freshness while maintaining energy sustainability. Future research will focus on relaxing the perfect Channel State Information (CSI) assumption and extending the framework to support mobile IoT devices and multi-cell environments with interfering IRSs.

Summary updated

BibTeX
@article{gong2023hierarchical,
  author = {Shimin Gong and Leiyang Cui and Bo Gu and Bin Lyu and Dinh Thai Hoang and Dusit Niyato},
  title = {{Hierarchical Deep Reinforcement Learning for Age-of-Information Minimization in IRS-aided and Wireless-powered Wireless Networks}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2023},
  url = {https://openreview.net/forum?id=sxf6rPfqzu},
}
Figure