Dynamic and Distributed Routing in IoT Networks based on Multi-Objective Q-Learning

Shubham Vaishnav, Praveen Kumar Donta, Sindri Magnússon

Introduction

Internet of Things (IoT) networks are characterized by diverse and often conflicting operational goals. For instance, a network might prioritize high reliability (Packet Delivery Ratio, PDR) during critical events while shifting to energy conservation during routine monitoring. Traditional routing protocols often struggle with these dynamic requirements, as most Reinforcement Learning (RL) approaches are either centralized—introducing single points of failure—or assume static objectives. When priorities shift, these systems typically require slow and energy-intensive retraining.

Comparison of DPQ-Learning with existing RL-based routing methods

This paper introduces Dynamic Preference Q-Learning (DPQ-Learning), a fully distributed routing protocol designed to bridge this gap. DPQ-Learning enables IoT devices to adapt to shifting priorities in real-time without retraining, maintaining near-optimal performance across the entire Pareto front of objective trade-offs.

Proposed Method: DPQ-Learning

The core of DPQ-Learning is a multi-objective framework that leverages off-policy RL to learn multiple policies simultaneously.

Parallel Per-Preference Learning

Instead of optimizing for a single fixed weight, each node maintains a grid of Q-tables, $\mathcal{B}$, representing different trade-offs between energy efficiency and PDR. Because Q-learning is an off-policy algorithm, a single packet transmission provides data that can update all Q-tables in the grid concurrently. This allows the network to build a comprehensive "knowledge base" of routing options for various preference vectors.

Distributed Architecture and Communication

The protocol is entirely decentralized. Each node $i$ manages its own local Q-tables. To minimize overhead, Q-value updates are "piggybacked" on standard acknowledgment (ACK) packets. This ensures that nodes learn about the network state through local interactions without requiring a central controller or additional signaling traffic.

Distributed Q-table architecture and packet flow in DPQ-Learning

Greedy Interpolation Policy (GIP)

A key challenge arises when the network encounters a preference weight $\beta$ that does not fall exactly on a grid point. To handle this, the authors developed the Greedy Interpolation Policy (GIP). Instead of selecting a sub-optimal grid point or retraining, GIP uses linear interpolation between the two closest learned Q-tables to derive a near-optimal action. The authors provide a theoretical proof that the optimal value function is Lipschitz-continuous with respect to preference parameters, guaranteeing that this interpolation remains close to the true optimal path.

Experimental Evaluation

The authors evaluated DPQ-Learning using a Python-based simulator modeling a 100-node grid network (IEEE 802.15.4). The protocol was compared against six baselines, including RL-based methods like RLBEEP, R2LTO, and RLBR, as well as a multi-objective RL variant, SMORLR.

Two primary scenarios were tested:

  1. Periodic: Objectives change every 1,000 episodes.
  2. Rapid: Objectives change every single episode, testing the limits of adaptation.

Key Results

  • Efficiency: DPQ-Learning reduced energy consumption by 80–90% compared to baseline protocols.
  • Performance: In dynamic settings, the method achieved up to 5$\times$ higher cumulative rewards and significantly improved PDR.
  • Instant Adaptation: Unlike the "brute force" SMORLR baseline, which suffered performance drops during preference shifts due to relearning, DPQ-Learning maintained high performance immediately following a change.
  • Robustness: Sensitivity analysis showed that DPQ-Learning remains superior even as the frequency of preference changes increases, whereas static-weight models converge to suboptimal "middle-ground" policies.

Conclusion and Future Work

DPQ-Learning demonstrates that distributed, multi-objective RL can effectively manage the trade-offs inherent in IoT routing. By using off-policy updates and greedy interpolation, it achieves instant adaptability with minimal communication overhead.

Future research directions include addressing the linear growth of memory requirements relative to grid points, which may challenge extremely resource-constrained devices. Additionally, the authors plan to extend the framework to higher-dimensional objective spaces and validate the protocol on hardware testbeds like FIT/IoT-LAB.

Summary updated

Abstract

IoT networks often face conflicting routing goals such as maximizing packet delivery, minimizing delay, and conserving limited battery energy. These priorities can also change dynamically: for example, an emergency alert requires high reliability, while routine monitoring prioritizes energy efficiency to prolong network lifetime. Existing works, including many deep reinforcement learning approaches, are typically centralized and assume static objectives, making them slow to adapt when preferences shift. We propose a dynamic and fully distributed multi-objective Q-learning routing algorithm that learns multiple per-preference Q-tables in parallel and introduces a novel greedy interpolation policy to act near-optimally for unseen preferences. The algorithm learns to optimize for energy efficiency, packet delivery ratio, and the composite reward, adapting to changing trade-offs between these metrics without retraining or centralized control. A theoretical analysis further shows that the optimal value function is Lipschitz-continuous in the preference parameter, ensuring that proposed greedy interpolation policy yields provably near-optimal behavior. Simulation results show that our approach adapts in real time to shifting priorities and achieves up to 80–90% lower energy consumption and up to 5 × higher cumulative rewards and packet delivery compared to six baseline protocols, under dynamic and distributed settings. Sensitivity analysis across varying preference window lengths confirms that the proposed DPQ framework consistently achieves higher composite reward than all baseline methods, demonstrating robustness to changes in operating conditions.
BibTeX
@misc{vaishnav2025dynamic,
  author = {Shubham Vaishnav and Praveen Kumar Donta and Sindri Magnússon},
  title = {{Dynamic and Distributed Routing in IoT Networks based on Multi-Objective Q-Learning}},
  year = {2025},
  eprint = {2505.00918},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.00918},
}
Figure