Autonomous Management of Energy-Harvesting IoT Nodes Using Deep Reinforcement Learning

Abdulmajid Murad, Frank Alexander Kraemer, Kerstin Bach, Gavin Taylor

TL;DR

This paper introduces a deep reinforcement learning (RL) approach, specifically Proximal Policy Optimization (PPO), to autonomously manage the energy consumption of wireless IoT nodes. By using PPO and a novel reward function that directly aligns with application goals, the system achieves more efficient energy usage and greater autonomy compared to traditional RL methods.

What Problem Does This Paper Address?

The proliferation of wireless IoT devices in smart cities, agriculture, and healthcare is hampered by their limited energy resources. Traditionally, managing these nodes' power consumption (e.g., duty cycles, sampling rates) relies on manual configuration or simplistic, pre-defined algorithms. This approach is insufficient for dynamic, non-stationary environments where nodes face unpredictable energy harvesting and varying operational demands. The paper addresses the need for autonomous IoT nodes that can learn to optimize their behavior in real-time, reducing manual design effort and enabling scalable IoT deployments.

Methods

The core of the paper's methodology lies in applying deep reinforcement learning, specifically the Proximal Policy Optimization (PPO) algorithm, to control IoT nodes. PPO is chosen for its ability to handle continuous action and observation spaces and its robust performance in complex domains. The authors utilize neural networks as function approximators to learn effective policies directly from continuous input data. A key innovation is the development of a new reward function that more closely reflects actual application goals, moving beyond simple energy neutrality metrics. They also propose training these RL agents on a server and updating the IoT nodes periodically, illustrated by a system setup involving a "Sensor Gym" simulator based on OpenAI Gym.

Datasets

The paper does not explicitly mention using pre-existing public datasets. Instead, it describes training and evaluating agents using simulated data. Specifically, they simulate harvested energy by calculating energy generated from solar radiation data, incorporating a 20% error to mimic real-world weather forecast inaccuracies. For training and evaluation, they use data from Tokyo for the years 2010 and 2011, respectively.

Key Insights and Ideas

  • Deep RL for IoT Autonomy: Deep RL, particularly PPO, is well-suited for the challenges of energy-harvesting IoT nodes, outperforming older RL techniques.
  • Beyond Energy Neutrality: Reward functions that directly align with application goals (e.g., maximizing data collection while minimizing variance and avoiding power failures) lead to more desirable and practical IoT node behavior than solely focusing on energy neutrality.
  • Continuous Action Spaces are Key: The ability of PPO to handle continuous action spaces (duty cycles) eliminates the need for manual discretization, simplifying the problem formulation and leading to more nuanced control.
  • Server-Side Training for Resource-Constrained Devices: Training complex RL agents on a server and deploying lightweight policies to IoT nodes is a practical approach for managing computational and energy constraints.
  • Hyperparameter Tuning Matters: The choice of hyperparameters, particularly the learning rate and the damping factor in the reward function, significantly influences the agent's behavior, allowing for trade-offs between energy utilization, variance, and power failures.

Key Results

  • PPO Outperforms SARSA: Agents trained with PPO consistently outperformed agents trained with SARSA(A) (an older RL technique) across various metrics, including energy utilization and deviation from energy neutrality. In one comparison, almost all PPO agents were better than the average SARSA agent.
  • Improved Reward Function Effectiveness: The new reward function, which balances duty cycle maximization with minimizing variance and power failures, led to more appropriate and continuous measurement coverage compared to reward functions based solely on energy neutrality.
  • Hyperparameter Impact: The study demonstrated that low learning rates resulted in agents that learned poorly, while appropriate damping factors in the reward function allowed for a balance between high energy utilization and acceptable variance and power failures. For instance, a damping factor of $\zeta=0.01$ yielded 95% utilized energy with 12 variance and 17 power failures over a year.

Limitations and Future Work

  • Simulation-Based Evaluation: The study relies on a simulated environment ("Sensor Gym"). While designed to be realistic, it may not fully capture all complexities of real-world IoT deployments.
  • Specific Application Focus: The reward function was designed for a specific application goal (maximizing duty cycle with minimal variance and avoiding failures). Future work could explore more complex and diverse application goals.
  • Hyperparameter Sensitivity: While the paper explores hyperparameter tuning, finding the optimal set for a given application might still require significant experimentation.
  • Future Work: The authors suggest that their work opens the door to solving even more complex problems and enabling broader applications of IoT in diverse domains by allowing engineers to focus on high-level goals rather than low-level technicalities. They believe there is potential for more complex reward functions and further exploration of hyperparameter optimization.

Summary updated

Abstract

Reinforcement learning (RL) is capable of managing wireless, energy-harvesting IoT nodes by solving the problem of autonomous management in non-stationary, resource-constrained settings. We show that the state-of-the-art policy-gradient approaches to RL are appropriate for the IoT domain and that they outperform previous approaches. Due to the ability to model continuous observation and action spaces, as well as improved function approximation capability, the new approaches are able to solve harder problems, permitting reward functions that are better aligned with the actual application goals. We show such a reward function and use policy-gradient approaches to learn capable policies, leading to behavior more appropriate for IoT nodes with less manual design effort, increasing the level of autonomy in IoT.
BibTeX
@inproceedings{murad2019autonomous,
  author = {Abdulmajid Murad and Frank Alexander Kraemer and Kerstin Bach and Gavin Taylor},
  title = {{Autonomous Management of Energy-Harvesting IoT Nodes Using Deep Reinforcement Learning}},
  booktitle = {SASO},
  year = {2019},
  url = {https://openreview.net/forum?id=iE0DIp0qLy},
}
Figure