This paper, recently submitted for review, introduces a novel framework for managing resources in complex, dynamic wireless environments. It specifically targets systems where Unmanned Aerial Vehicles (UAVs) and Intelligent Reflecting Surfaces (IRS) work together to support Mobile Edge Computing (MEC).
TL;DR
The authors propose FRES (Flexible REsource Scheduling), a deep progressive reinforcement learning framework that minimizes energy consumption in MEC systems. By using a "progressive" neural architecture, the system can adapt to a changing number of UAVs in real-time without "forgetting" previous configurations or requiring total retraining.
What Problem Does This Paper Address?
Mobile Edge Computing (MEC) allows power-hungry devices (like VR headsets or IoT sensors) to offload heavy computations to nearby servers. Using UAVs as mobile servers adds flexibility, and IRS (smart mirrors for radio waves) helps bypass obstacles like buildings.
However, two major hurdles exist:
- The "Moving Target" Problem: Most existing models assume a fixed number of UAVs. In reality, UAVs may enter or leave the service area. Standard Deep Reinforcement Learning (DRL) models struggle here; if you change the network structure to accommodate a new UAV, the model often suffers from "catastrophic forgetting," losing its previous optimization logic.
- Complexity: Scheduling involves both discrete decisions (which UAV should I connect to?) and continuous variables (how much power/bandwidth should be allocated?). This creates a Mixed Integer Nonlinear Programming (MINLP) problem, which is notoriously difficult to solve quickly.
Methods
The researchers developed the Flexible REsource Scheduling (FRES) framework, which relies on three main pillars:
- Multi-task Agent: Instead of separate models, a single agent uses a shared "backbone" to process environment data. It then splits into two "heads": a classification head for discrete offloading decisions and a fitting head for continuous resource allocation.
- Progressive Scheduler: This is the core innovation. When the number of UAVs changes, the framework doesn't scrap the model. Instead, it "progressively" adds or removes specific neurons. This allows the agent to accumulate experience over time and switch between different UAV counts almost instantly.
- Light Taboo Search (LTS): Reinforcement learning can sometimes get stuck in "local minima" (good-but-not-great solutions). The authors added a lightweight version of Taboo Search—a mathematical local search algorithm—to refine the DRL's output and ensure the system finds a near-optimal global solution.
Datasets
The study relies on a simulated MEC environment modeled after a bustling urban center. Key parameters include:
- Scale: 50 User Equipments (UEs) and up to 5 UAVs.
- Task Specs: Computation tasks ranging from 19–21 MB with CPU requirements around $10^9$ cycles/s.
- Hardware Simulation: Modeled on an Intel Xeon CPU with a Tesla T4 GPU.
- Comparison Baselines: The authors compared FRES against established DRL algorithms including DDPG, SAC, LyDROO, and ARE.
Key Insights and Ideas
- Immunity to Catastrophic Forgetting: By using a progressive structure, the model treats a change in UAV count as an expansion of knowledge rather than a replacement of it. When a UAV leaves and then returns, the model can recall its previous "old task" parameters in just 0.43 seconds, compared to over 50 seconds for traditional models that must retrain.
- Hybrid Intelligence: The paper demonstrates that DRL is best used for the "heavy lifting" of global patterns, while classical search heuristics (like LTS) are excellent for fine-tuning the final answer.
- IRS as a Power Saver: The integration of IRS phase-shift optimization significantly improves signal quality, which in turn reduces the transmission power required by mobile devices, directly lowering the total system energy footprint.
Key Results
- Energy Efficiency: FRES achieved an average energy consumption of 56.37 units, outperforming DDPG (61.55) and SAC (60.09).
- Adaptation Speed: In scenarios where the number of UAVs fluctuated (e.g., 3 UAVs $\rightarrow$ 4 UAVs $\rightarrow$ 3 UAVs), FRES maintained a stable reward curve, whereas competing models saw significant performance dips and required lengthy recovery periods.
- Search Quality: The Light Taboo Search (LTS) refinement allowed the framework to converge to a lower energy state faster than other exploration methods like Simulated Annealing.
Limitations and Future Work
While the progressive architecture is efficient, the authors note a potential "parameter bloat" issue. As the number of UAVs and tasks increases, the number of neurons in the multi-task agent grows.
Future Work suggested by the authors includes:
- Implementing model pruning and quantization to keep the neural network lean as it grows.
- Exploring compression techniques for the network structure to ensure it can run on resource-constrained hardware within the UAVs themselves.
Paper Citation: Li Dong, et al. "Deep progressive reinforcement learning-based flexible resource scheduling framework for IRS and UAV-assisted MEC system." arXiv:2408.01248v1 [cs.LG] 2 Aug 2024.
Summary updated