Secure Resource Allocation via Constrained Deep Reinforcement Learning

Jianfei Sun, Qiang Gao, Cong Wu, Yuxian Li, Jiacheng Wang, Dusit Niyato

TL;DR

The SARMTO framework introduces a novel "action-constrained" deep reinforcement learning approach to manage resource allocation in serverless multi-cloud edge environments. By embedding system constraints directly into the learning architecture rather than relying on simple penalties, it achieves up to a 40% reduction in system costs and a 41.5% boost in energy efficiency compared to traditional methods.

What Problem Does This Paper Address?

As we move toward 6G and the proliferation of IoT, applications like augmented reality and autonomous driving generate tasks far too heavy for local devices to process. While "serverless multi-cloud edge computing" offers a solution by offloading these tasks to nearby edge nodes or distant clouds, managing this environment is incredibly difficult.

The authors identify three major gaps in current research:

  1. Complexity of Heterogeneity: Most models assume a single cloud/edge setup, ignoring the reality of diverse nodes with different speeds, costs, and latencies.
  2. The "Penalty" Problem: Traditional Deep Reinforcement Learning (DRL) handles system constraints (like maximum allowable delay) by giving the agent a "penalty" (negative reward) when it fails. This often leads to slow convergence and suboptimal "safe" policies.
  3. Security as an Afterthought: Security protocols (encryption/decryption) consume significant CPU cycles and energy, yet they are rarely integrated into the resource optimization loop.

Methods

The core of the paper is SARMTO (Secure Adaptive Resource Management and Task Optimization), which utilizes a specialized algorithm called AC-DQN (Action-Constrained Deep Q-Network).

Key technical components include:

  • Action-Constrained Learning: Unlike standard DRL, AC-DQN incorporates a constraint function $f_{constraint}$ directly into the target Q-value calculation. This effectively "shapes" the mathematical landscape the agent explores, teaching it to inherently avoid infeasible actions (like offloading a massive task to a slow node that would violate delay constraints).
  • Integrated Security Model: The framework treats security as a resource-consuming task. It models the CPU cycles required for RSA encryption/decryption and MD5 integrity verification, allowing the agent to factor "security overhead" into its offloading decisions.
  • Dueling Network Architecture: The neural network splits into two streams—one to estimate the value of being in a certain state and another to estimate the advantage of specific actions. This helps the agent learn which states are valuable even when actions don't have an immediate impact.
  • Prioritized Experience Replay (PER): Instead of learning from random past experiences, the agent focuses on "surprising" transitions (those with high temporal-difference errors), speeding up the learning process.

Datasets

The researchers evaluated SARMTO using extensive simulations based on real-world application profiles. They utilized a task model with seven distinct application types, each with specific "CPU cycles per byte" requirements:

  • Low Complexity: Gzip (330 cycles/byte), Health monitoring (500).
  • Medium Complexity: pdf2text (960), x264 CBR encode (1900).
  • High Complexity: html2text (5900), Augmented Reality (12000).

The simulation environment (Python 3.9) modeled a network with one MEC node (10 GHz) and two Cloud nodes (100 GHz), with task data sizes ranging from 1 GB to 5 GB.

Key Insights and Ideas

  • Constraints as Architecture, Not Just Rewards: The most significant "aha moment" is that embedding constraints into the value function calculation is far more stable than using reward penalties. It prevents the agent from "wandering" into invalid solution spaces during training.
  • The Security-Energy Trade-off: Security isn't just a binary "on/off" feature; it's a resource drain. SARMTO demonstrates that an agent can learn to balance the energy cost of encryption against the latency benefits of offloading.
  • Adaptive Offloading: The agent learns a nuanced strategy: at low edge capacities, it offloads heavily to the cloud (up to 78.3%); as edge capacity increases, it intelligently shifts tasks back to the edge (down to 41.2%) to save on communication energy.

Key Results

SARMTO was tested against five baselines, including standard DQN and "Local-only" processing.

  • Cost Reduction: At high workloads (1000 tasks), SARMTO reduced total system costs by 23.6% compared to standard DQN and 40% compared to random cloud offloading.
  • Energy Efficiency: For large data tasks (5 GB), SARMTO consumed 24.1% less energy than standard DQN and 41.5% less than random offloading.
  • Latency Balance: While SARMTO wasn't always the fastest (random cloud offloading is sometimes faster but extremely energy-expensive), it maintained competitive delays while achieving much higher overall efficiency.

Limitations and Future Work

The authors acknowledge that while their simulation is robust, it remains a simulation. Real-world network jitter and unpredictable hardware failures in a live multi-cloud environment might challenge the model's stability.

Future directions include: * Scaling the framework to handle the massive resource demands of Large Language Models (LLMs) at the edge. * Extending the model to more complex, dynamic security threat landscapes. * Applying the framework to distributed learning systems (like Federated Learning) where resource allocation is a primary bottleneck.

Summary updated

Abstract

The proliferation of Internet of Things (IoT) devices and the advent of 6G technologies have introduced computationally intensive tasks that often surpass the processing capabilities of user devices. Efficient and secure resource allocation in serverless multi-cloud edge computing environments is essential for supporting these demands and advancing distributed computing. However, existing solutions frequently struggle with the complexity of multi-cloud infrastructures, robust security integration, and effective application of traditional deep reinforcement learning (DRL) techniques under system constraints. To address these challenges, we present SARMTO, a novel framework that integrates an action-constrained DRL model. SARMTO dynamically balances resource allocation, task offloading, security, and performance by utilizing a Markov decision process formulation, an adaptive security mechanism, and sophisticated optimization techniques. Extensive simulations across varying scenarios, including different task loads, data sizes, and MEC capacities, show that SARMTO consistently outperforms five baseline approaches, achieving up to a 40% reduction in system costs and a 41.5% improvement in energy efficiency over state-of-the-art methods. These enhancements highlight SARMTO's potential to revolutionize resource management in intricate distributed computing environments, opening the door to more efficient and secure IoT and edge computing applications.
BibTeX
@inproceedings{sun2024secure,
  author = {Jianfei Sun and Qiang Gao and Cong Wu and Yuxian Li and Jiacheng Wang and Dusit Niyato},
  title = {{Secure Resource Allocation via Constrained Deep Reinforcement Learning}},
  booktitle = {Ml4cs},
  year = {2024},
  doi = {10.1007/978-981-96-4566-4\_1},
}
Figure