Secure Resource Allocation via Constrained Deep Reinforcement Learning

Jianfei Sun, Qiang Gao, Cong Wu, Yuxian Li, Jiacheng Wang, Dusit Niyato

Introduction

The advent of 6G technology and the exponential growth of Internet of Things (IoT) devices have ushered in a new era of data-intensive applications. Technologies such as augmented reality (AR), autonomous driving, and real-time health monitoring generate massive computational tasks that far exceed the processing capabilities of local mobile devices. To address this, serverless multi-cloud edge computing has emerged as a promising paradigm, allowing tasks to be offloaded to nearby Multi-access Edge Computing (MEC) nodes or more powerful, distant cloud centers. However, managing resource allocation in such a heterogeneous and dynamic environment presents significant challenges, particularly regarding latency, energy consumption, and security.

System Architecture

The proposed SARMTO (Secure Adaptive Resource Management and Task Optimization) framework operates within a multi-tiered network architecture. This environment consists of diverse user devices, local MEC nodes, and remote serverless cloud computing (CC) nodes.

System architecture of the serverless multi-cloud edge computing environment.

As illustrated in Figure 1, the system must navigate a complex web of communication paths. The primary goal is to determine the optimal offloading strategy—deciding whether a task should be processed locally, at an edge node, or in the cloud—while accounting for varying node speeds, communication costs, and strict latency requirements.

The SARMTO Framework

Traditional Deep Reinforcement Learning (DRL) approaches often struggle with hard system constraints. Typically, these constraints (such as maximum allowable delay) are handled via penalty functions, where the agent receives a negative reward for violations. This often results in slow convergence and a tendency for the agent to adopt overly "safe" but suboptimal policies.

SARMTO introduces the Action-Constrained Deep Q-Network (AC-DQN) to overcome these limitations. Instead of relying solely on rewards, AC-DQN embeds a constraint function directly into the target Q-value calculation. This architectural change shapes the action space, guiding the agent to inherently avoid infeasible actions that would violate system boundaries. Furthermore, the framework utilizes a Dueling Network architecture to separate state-value estimation from action-advantage estimation, and Prioritized Experience Replay (PER) to focus learning on the most informative transitions.

Integrated Security and Resource Management

A key innovation of SARMTO is the integration of security as a first-class resource consideration. Security protocols like RSA for encryption/decryption and MD5 for integrity verification are not "free"; they consume significant CPU cycles and energy. SARMTO models these overheads, allowing the agent to perform a nuanced trade-off analysis. It balances the energy cost of securing a task against the latency benefits of offloading it to a more powerful remote node.

Experimental Evaluation

The researchers evaluated SARMTO through extensive simulations, modeling a network with one 10 GHz MEC node and two 100 GHz Cloud nodes. The task model included seven distinct application types with varying computational intensities.

Computational complexity profiles for various application types used in the simulation.

As shown in Figure 2, the applications range from low-complexity tasks like Gzip compression (330 cycles/byte) to high-complexity tasks like Augmented Reality (12,000 cycles/byte). This diversity ensures the model is tested against a realistic spectrum of workloads.

Key Results

SARMTO demonstrated superior performance across several metrics compared to standard DQN and "Local-only" baselines:

  • Cost Reduction: In high-load scenarios (1,000 tasks), SARMTO reduced total system costs by 23.6% over standard DQN and 40% compared to random cloud offloading.
  • Energy Efficiency: For large-scale tasks (5 GB), the framework achieved a 24.1% reduction in energy consumption compared to standard DQN and a 41.5% reduction compared to random offloading.
  • Adaptive Strategy: The agent successfully learned to shift offloading targets based on edge capacity, moving from 78.3% cloud offloading at low capacity to 41.2% as edge resources became more available.

Conclusion and Future Work

The SARMTO framework represents a significant step forward in secure, constraint-aware resource management for edge-cloud environments. By embedding constraints into the learning architecture and treating security as a measurable resource drain, it achieves substantial gains in efficiency and cost-effectiveness.

Future research directions include scaling this approach for the massive resource demands of Large Language Models (LLMs) at the edge and applying the framework to distributed learning paradigms like Federated Learning, where resource allocation remains a primary bottleneck.

Summary updated

BibTeX
@inproceedings{sun2024secure,
  author = {Jianfei Sun and Qiang Gao and Cong Wu and Yuxian Li and Jiacheng Wang and Dusit Niyato},
  title = {{Secure Resource Allocation via Constrained Deep Reinforcement Learning}},
  booktitle = {Ml4cs},
  year = {2024},
  doi = {10.1007/978-981-96-4566-4\_1},
}
Figure