Differentially Private Deep Q-Learning for Pattern Privacy Preservation in MEC Offloading

Shuying Gan, Marie Siew, Chao Xu, Tony Q. S. Quek

Abstract

Mobile Edge Computing (MEC) enhances the capabilities of resource-constrained IoT devices by offloading tasks to edge or cloud servers. However, the decision-making process itself can leak sensitive information regarding user habits and server states, a vulnerability known as Pattern Privacy (PP) leakage. This paper presents DP-DQO, a novel deep reinforcement learning framework that integrates a Function-output Gaussian Process Mechanism (FGPM) into a Deep Q-Network (DQN). By injecting calibrated noise into the network's output, DP-DQO provides rigorous $(\epsilon, \delta)$-differential privacy guarantees for offloading patterns without the energy overhead of traditional dummy-task methods. Simulations demonstrate that DP-DQO achieves a robust balance between system utility (latency and energy consumption) and privacy preservation.

1. Introduction

As IoT devices become ubiquitous, Mobile Edge Computing (MEC) has emerged as a vital architecture for processing computation-intensive tasks. While much research has focused on protecting the content of offloaded data, the patterns of offloading—when, where, and how often a device offloads—remain a significant security blind spot. An adversary monitoring these decisions can employ inverse reinforcement learning (IRL) to reconstruct sensitive metadata, such as a user’s daily routine or the real-time queue status of an edge server.

Existing countermeasures often rely on generating "dummy" tasks to obfuscate real traffic. While effective at masking patterns, these methods significantly increase energy consumption and bandwidth usage. This paper proposes a more efficient alternative: protecting privacy through the stochasticity of the decision-making algorithm itself rather than through redundant data transmission.

2. Problem Formulation

The authors model the offloading environment as a Markov Decision Process (MDP). The goal is to optimize a multi-objective cost function that includes:

  • Latency: The time required for task transmission and computation.
  • Energy Consumption: The power used by the mobile device for local processing or wireless transmission.
  • Task Dropping Rate: The frequency of tasks failing to meet deadlines.

In this MDP, the state space includes the current task characteristics and server queue lengths, while the action space consists of deciding whether to process a task locally, at the edge server (ES), or at the cloud server (CS).

3. The DP-DQO Algorithm

The core of the proposed solution is the Differential Privacy Deep Q-learning based Offloading (DP-DQO) algorithm. It builds upon the standard DQN architecture but introduces a specialized privacy layer.

3.1 Function-output Gaussian Process Mechanism (FGPM)

To prevent an observer from mapping specific offloading actions back to exact system states, the authors utilize FGPM. Unlike standard differential privacy which might add noise to the input data or the gradients, FGPM injects noise directly into the Q-values (the predicted rewards for each action) produced by the DQN's output layer.

This noise is sampled from a Gaussian process, ensuring that "adjacent" state-action pairs—those that yield similar rewards—become indistinguishable to an external observer. This provides a mathematically rigorous $(\epsilon, \delta)$-differential privacy guarantee, ensuring that the probability of an attacker correctly guessing the system state from an observed action is strictly bounded.

3.2 Theoretical Guarantees

The paper provides two critical theoretical proofs:

  1. Privacy Guarantee: A derivation of the privacy budget $\epsilon$ relative to the noise scale $\sigma$.
  2. Utility Guarantee: A proof that the learning error (the performance gap between DP-DQO and a non-private optimal policy) remains bounded. As the state space grows, the relative impact of the noise diminishes, allowing the algorithm to maintain high utility in complex environments.

4. Experimental Evaluation

The researchers evaluated DP-DQO using numerical simulations involving five mobile devices, one edge server, and one cloud server. Task arrivals were modeled as a Poisson process, with data sizes ranging from 5 to 50 MB.

4.1 The Privacy-Utility Trade-off

The experiments highlight a clear trade-off: as the noise level ($\sigma$) increases, the privacy protection becomes stronger, but the cumulative "Return" (efficiency) decreases. High noise levels introduce instability during the training phase, requiring more episodes to reach convergence. However, at moderate noise levels (e.g., $\sigma = 0.1$), the algorithm performs nearly as well as a standard, non-private DQN.

4.2 Comparative Performance

DP-DQO was compared against a "Greedy" baseline, which always chooses the action with the lowest immediate cost. In all scenarios, the reinforcement learning approach significantly outperformed the greedy heuristic, demonstrating that the agent successfully learns long-term strategies even when its observations are perturbed by privacy-preserving noise.

5. Conclusion and Future Work

DP-DQO represents a significant step toward efficient privacy in MEC by moving away from resource-heavy dummy tasks. By embedding privacy directly into the reinforcement learning output, the system protects sensitive usage patterns with minimal impact on device battery life or network latency.

Future research directions include:

  • Dynamic Network Models: Incorporating fluctuating wireless channel conditions into the simulation.
  • Scalability: Testing the framework in massive IoT deployments with hundreds of concurrent devices.
  • Empirical Attack Testing: Evaluating the algorithm against actual Inverse Reinforcement Learning (IRL) attacks to quantify the practical difficulty of information recovery for an adversary.

Summary updated

BibTeX
@inproceedings{gan2023differentially,
  author = {Shuying Gan and Marie Siew and Chao Xu and Tony Q. S. Quek},
  title = {{Differentially Private Deep Q-Learning for Pattern Privacy Preservation in MEC Offloading}},
  booktitle = {ICC},
  year = {2023},
  doi = {10.1109/icc45041.2023.10278840},
}
Figure