Introduction
Internet of Things (IoT) networks are characterized by diverse and often conflicting operational goals. For instance, a network might prioritize high reliability (Packet Delivery Ratio, PDR) during critical events while shifting to energy conservation during routine monitoring. Traditional routing protocols often struggle with these dynamic requirements, as most Reinforcement Learning (RL) approaches are either centralized—introducing single points of failure—or assume static objectives. When priorities shift, these systems typically require slow and energy-intensive retraining.
This paper introduces Dynamic Preference Q-Learning (DPQ-Learning), a fully distributed routing protocol designed to bridge this gap. DPQ-Learning enables IoT devices to adapt to shifting priorities in real-time without retraining, maintaining near-optimal performance across the entire Pareto front of objective trade-offs.
Proposed Method: DPQ-Learning
The core of DPQ-Learning is a multi-objective framework that leverages off-policy RL to learn multiple policies simultaneously.
Parallel Per-Preference Learning
Instead of optimizing for a single fixed weight, each node maintains a grid of Q-tables, $\mathcal{B}$, representing different trade-offs between energy efficiency and PDR. Because Q-learning is an off-policy algorithm, a single packet transmission provides data that can update all Q-tables in the grid concurrently. This allows the network to build a comprehensive "knowledge base" of routing options for various preference vectors.
Distributed Architecture and Communication
The protocol is entirely decentralized. Each node $i$ manages its own local Q-tables. To minimize overhead, Q-value updates are "piggybacked" on standard acknowledgment (ACK) packets. This ensures that nodes learn about the network state through local interactions without requiring a central controller or additional signaling traffic.
Greedy Interpolation Policy (GIP)
A key challenge arises when the network encounters a preference weight $\beta$ that does not fall exactly on a grid point. To handle this, the authors developed the Greedy Interpolation Policy (GIP). Instead of selecting a sub-optimal grid point or retraining, GIP uses linear interpolation between the two closest learned Q-tables to derive a near-optimal action. The authors provide a theoretical proof that the optimal value function is Lipschitz-continuous with respect to preference parameters, guaranteeing that this interpolation remains close to the true optimal path.
Experimental Evaluation
The authors evaluated DPQ-Learning using a Python-based simulator modeling a 100-node grid network (IEEE 802.15.4). The protocol was compared against six baselines, including RL-based methods like RLBEEP, R2LTO, and RLBR, as well as a multi-objective RL variant, SMORLR.
Two primary scenarios were tested:
- Periodic: Objectives change every 1,000 episodes.
- Rapid: Objectives change every single episode, testing the limits of adaptation.
Key Results
- Efficiency: DPQ-Learning reduced energy consumption by 80–90% compared to baseline protocols.
- Performance: In dynamic settings, the method achieved up to 5$\times$ higher cumulative rewards and significantly improved PDR.
- Instant Adaptation: Unlike the "brute force" SMORLR baseline, which suffered performance drops during preference shifts due to relearning, DPQ-Learning maintained high performance immediately following a change.
- Robustness: Sensitivity analysis showed that DPQ-Learning remains superior even as the frequency of preference changes increases, whereas static-weight models converge to suboptimal "middle-ground" policies.
Conclusion and Future Work
DPQ-Learning demonstrates that distributed, multi-objective RL can effectively manage the trade-offs inherent in IoT routing. By using off-policy updates and greedy interpolation, it achieves instant adaptability with minimal communication overhead.
Future research directions include addressing the linear growth of memory requirements relative to grid points, which may challenge extremely resource-constrained devices. Additionally, the authors plan to extend the framework to higher-dimensional objective spaces and validate the protocol on hardware testbeds like FIT/IoT-LAB.
Summary updated