Introduction
In modern Internet of Things (IoT) networks, the value of data is often tied to its "freshness." Applications such as environmental monitoring or real-time surveillance rely on timely updates. This freshness is quantified by the Age of Information (AoI), defined as the time elapsed since the last successfully received update was generated. Minimizing AoI is challenging due to the inherent energy-freshness tradeoff: frequent data processing keeps information fresh but rapidly depletes the limited battery life of IoT devices. Furthermore, IoT environments are often non-stationary, with data arrival rates and packet sizes changing over time, rendering static reinforcement learning (RL) models ineffective.
System Model and AoI Dynamics
The system considers a set of IoT devices tasked with processing and transmitting data to a central hub. The evolution of AoI for a device $i$ at time $t$, denoted as $\Delta_{i,t}$, is characterized by a sawtooth pattern.
As shown in the figure above, the AoI increases linearly during idle periods. When a device begins processing a task at time $u_{i,k}$ and completes it at $v_{i,k}$, the AoI drops to the age of the newly delivered packet. Effective management requires balancing these processing intervals against energy consumption in unpredictable environments.
Proposed Methodology: Lifelong Reinforcement Learning
To address environmental non-stationarity, this paper proposes a Lifelong Reinforcement Learning (LLRL) framework. The core innovation is the use of an Unmanned Aerial Vehicle (UAV) as a mobile "knowledge broker."
Instead of learning each new task from scratch, the UAV collects experience trajectories from various devices and maintains a global Knowledge Base ($L$) of latent policy components. A device's specific policy for a new task is represented as a sparse linear combination of these shared components. This approach allows for a "warm start," where a device can immediately adopt a near-optimal policy by identifying relevant past experiences. To prevent "catastrophic forgetting," the knowledge base is updated incrementally using second-order Taylor expansions, ensuring the system evolves without needing to store all historical data.
Experimental Results
The framework was evaluated using simulations based on the Mica2 Platform wireless sensor node. Ten distinct tasks were generated by varying data arrival rates and packet sizes. The LLRL algorithm was compared against a standard Policy Gradient baseline.
As illustrated in the performance plots, the proposed LLRL algorithm consistently achieves 25% to 50% faster convergence than the baseline. Key observations include:
- Superior Initial Performance: The "warm start" capability provides a significantly higher initial reward (approximately 10% higher) compared to random initialization.
- Global Optima: In complex scenarios (e.g., plot d), the baseline often gets trapped in local optima, while the LLRL method leverages accumulated knowledge to find the global optimum.
- Overall Efficiency: The algorithm achieved an 8% higher mean average reward across all sequential tasks.
Conclusion and Future Work
The LLRL framework significantly improves the adaptability and efficiency of IoT networks in dynamic environments. By treating UAVs as intelligence hubs rather than just relays, the system minimizes AoI while preserving energy. Future research will focus on optimizing UAV flight trajectories to maximize knowledge distribution and exploring correlations between environmental tasks to further enhance transfer learning efficiency.
Summary updated