Lifelong Learning for Minimizing Age of Information in Internet of Things Networks

Zhenzhen Gong, Qimei Cui, Christina Chaccour, Bo Zhou, Mingzhe Chen, Walid Saad

Introduction

In modern Internet of Things (IoT) networks, the value of data is often tied to its "freshness." Applications such as environmental monitoring or real-time surveillance rely on timely updates. This freshness is quantified by the Age of Information (AoI), defined as the time elapsed since the last successfully received update was generated. Minimizing AoI is challenging due to the inherent energy-freshness tradeoff: frequent data processing keeps information fresh but rapidly depletes the limited battery life of IoT devices. Furthermore, IoT environments are often non-stationary, with data arrival rates and packet sizes changing over time, rendering static reinforcement learning (RL) models ineffective.

System Model and AoI Dynamics

The system considers a set of IoT devices tasked with processing and transmitting data to a central hub. The evolution of AoI for a device $i$ at time $t$, denoted as $\Delta_{i,t}$, is characterized by a sawtooth pattern.

Evolution of Age of Information (AoI) over time, showing idle and processing intervals.

As shown in the figure above, the AoI increases linearly during idle periods. When a device begins processing a task at time $u_{i,k}$ and completes it at $v_{i,k}$, the AoI drops to the age of the newly delivered packet. Effective management requires balancing these processing intervals against energy consumption in unpredictable environments.

Proposed Methodology: Lifelong Reinforcement Learning

To address environmental non-stationarity, this paper proposes a Lifelong Reinforcement Learning (LLRL) framework. The core innovation is the use of an Unmanned Aerial Vehicle (UAV) as a mobile "knowledge broker."

Instead of learning each new task from scratch, the UAV collects experience trajectories from various devices and maintains a global Knowledge Base ($L$) of latent policy components. A device's specific policy for a new task is represented as a sparse linear combination of these shared components. This approach allows for a "warm start," where a device can immediately adopt a near-optimal policy by identifying relevant past experiences. To prevent "catastrophic forgetting," the knowledge base is updated incrementally using second-order Taylor expansions, ensuring the system evolves without needing to store all historical data.

Experimental Results

The framework was evaluated using simulations based on the Mica2 Platform wireless sensor node. Ten distinct tasks were generated by varying data arrival rates and packet sizes. The LLRL algorithm was compared against a standard Policy Gradient baseline.

Comparison of Average Reward vs. Iterations for the proposed LLRL algorithm and a standard Policy Gradient baseline across four different tasks (a-d).

As illustrated in the performance plots, the proposed LLRL algorithm consistently achieves 25% to 50% faster convergence than the baseline. Key observations include:

  • Superior Initial Performance: The "warm start" capability provides a significantly higher initial reward (approximately 10% higher) compared to random initialization.
  • Global Optima: In complex scenarios (e.g., plot d), the baseline often gets trapped in local optima, while the LLRL method leverages accumulated knowledge to find the global optimum.
  • Overall Efficiency: The algorithm achieved an 8% higher mean average reward across all sequential tasks.

Conclusion and Future Work

The LLRL framework significantly improves the adaptability and efficiency of IoT networks in dynamic environments. By treating UAVs as intelligence hubs rather than just relays, the system minimizes AoI while preserving energy. Future research will focus on optimizing UAV flight trajectories to maximize knowledge distribution and exploring correlations between environmental tasks to further enhance transfer learning efficiency.

Summary updated

Abstract

In this paper, a lifelong learning problem is studied for an Internet of Things (IoT) system. In the considered model, each IoT device aims to balance its information freshness and energy consumption tradeoff by controlling its computational resource allocation at each time slot under dynamic environments. An unmanned aerial vehicle (UAV) is deployed as a flying base station so as to enable the IoT devices to adapt to novel environments. To this end, a new lifelong reinforcement learning algorithm, used by the UAV, is proposed in order to adapt the operation of the devices at each visit by the UAV. By using the experience from previously visited devices and environments, the UAV can help devices adapt faster to future states of their environment. To do so, a knowledge base shared by all devices is maintained at the UAV. Simulation results show that the proposed algorithm can converge 25% to 50% faster than a policy gradient baseline algorithm that optimizes each device’s decision making problem in isolation.
BibTeX
@inproceedings{gong2021lifelong,
  author = {Zhenzhen Gong and Qimei Cui and Christina Chaccour and Bo Zhou and Mingzhe Chen and Walid Saad},
  title = {{Lifelong Learning for Minimizing Age of Information in Internet of Things Networks}},
  booktitle = {ICC},
  year = {2021},
  url = {https://openreview.net/forum?id=eJsSr24kIx7},
}
Figure