Reinforcement Learning for Minimizing Age of Information in Real-time Internet of Things Systems with Realistic Physical Dynamics

Sihua Wang, Mingzhe Chen, Zhaohui Yang, Changchuan Yin, Walid Saad, Shuguang Cui, H. Vincent PoorarXiv 2021 · OpenReview

Abstract

In Internet of Things (IoT) systems, maintaining data freshness is critical for real-time monitoring and control. This paper addresses the challenge of minimizing the Age of Information (AoI) while considering the energy constraints of battery-powered devices and the nonlinear dynamics of physical processes. We propose a distributed reinforcement learning (RL) framework based on the QMIX algorithm. By integrating physics-aware sampling policies with a centralized-training-decentralized-execution architecture, the system optimizes both sampling at the device level and resource allocation at the base station. Experimental results using real-world pollution data demonstrate that our approach reduces AoI by up to 33.9% and energy consumption by 35.1% compared to traditional uniform sampling methods.

Introduction

The proliferation of IoT devices has enabled real-time monitoring of diverse physical processes, from environmental sensing to industrial automation. In these applications, the value of data is often tied to its "freshness," quantified by the Age of Information (AoI). However, achieving low AoI is complicated by two primary factors: the limited energy reserves of IoT devices and the scarcity of wireless communication resources. Most existing research assumes simplistic periodic sampling, which fails to capture the volatile, nonlinear nature of real-world physical dynamics. This paper introduces a framework that adapts sampling and transmission strategies to the underlying physics of the monitored environment.

Methodology

The proposed solution treats the problem as a joint optimization of the Sampling Policy at the IoT device and the Device Selection at the Base Station (BS).

Physics-Aware AoI Modeling

Unlike standard AoI metrics that rely solely on time elapsed, this model incorporates the dynamics of the physical process. Using Nyquist-Shannon sampling theory, the authors derive a relationship between the variation frequency of the physical signal and the required sampling rate. If a device samples at a rate lower than the signal's Nyquist frequency, the AoI is penalized more heavily to reflect the increased estimation error and loss of tracking accuracy.

Distributed QMIX Framework

To manage the coordination between multiple devices and the BS, the researchers employ the QMIX multi-agent RL algorithm. The architecture follows a centralized-training-decentralized-execution (CTDE) paradigm:

  • Local Agents: Each IoT device operates a local Deep Q-Network (DQN) to make sampling decisions based on local observations of its energy state and the physical process.
  • Mixing Network: During training, a mixing network at the BS aggregates local Q-values into a global joint action-value function. This ensures that devices learn cooperative behaviors, such as prioritizing transmissions for devices with the most critical data updates.
System architecture of the multi-agent RL framework for joint sampling and device selection.

Experimental Results

The framework was evaluated using a real-world dataset of PM 2.5 air pollution levels from Peking University. This dataset provides a challenging, nonlinear testbed for tracking accuracy, capturing the complex fluctuations of air quality in Beijing.

Performance Comparison

The QMIX-based approach was compared against standard DQN and uniform periodic sampling. As shown in the convergence plots, the QMIX framework achieves higher rewards and faster convergence by effectively managing the trade-off between data freshness and energy expenditure.

Reward convergence comparison between QMIX and standard DQN methods.

Key performance metrics include:

  • AoI Reduction: The system achieved a 17.8% reduction in total AoI compared to standard DQN and a 33.9% reduction compared to uniform sampling.
  • Energy Efficiency: Total energy consumption was reduced by 13.2% relative to DQN and 35.1% relative to uniform sampling.
  • Tracking Accuracy: The QMIX approach demonstrated superior reconstruction of the original PM 2.5 signal, accurately capturing pollution spikes that were missed by other methods.

Conclusion

By explicitly modeling physical dynamics and employing a cooperative multi-agent RL framework, this research provides a robust solution for real-time IoT monitoring. The QMIX-based approach successfully balances the competing demands of data freshness and energy conservation. Future work will explore the scalability of this framework in massive IoT networks and its application to mobile base stations, such as those mounted on unmanned aerial vehicles (UAVs).

Summary updated

BibTeX
@misc{wang2021reinforcement,
  author = {Sihua Wang and Mingzhe Chen and Zhaohui Yang and Changchuan Yin and Walid Saad and Shuguang Cui and H. Vincent Poor},
  title = {{Reinforcement Learning for Minimizing Age of Information in Real-time Internet of Things Systems with Realistic Physical Dynamics}},
  year = {2021},
  url = {https://openreview.net/forum?id=01FRrHMgnF},
}
Figure