A Federated Reinforcement Learning Method with Quantization for Cooperative Edge Caching in Fog Radio Access Networks

Yanxiang Jiang, Min Zhang, Fu‐Chun Zheng, Yan Chen, Mehdi Bennis, Xiaohu YouarXiv 2022 · DOI · arXiv

Abstract

In this paper, cooperative edge caching problem is studied in fog radio access networks (F-RANs). Given the non-deterministic polynomial hard (NP-hard) property of the problem, a dueling deep Q network (Dueling DQN) based caching update algorithm is proposed to make an optimal caching decision by learning the dynamic network environment. In order to protect user data privacy and solve the problem of slow convergence of the single deep reinforcement learning (DRL) model training, we propose a federated reinforcement learning method with quantization (FRLQ) to implement cooperative training of models from multiple fog access points (F-APs) in F-RANs. To address the excessive consumption of communications resources caused by model transmission, we prune and quantize the shared DRL models to reduce the number of model transfer parameters. The communications interval is increased and the communications rounds are reduced by periodical model global aggregation. We analyze the global convergence and computational complexity of our policy. Simulation results verify that our policy has better performance in reducing user request delay and improving cache hit rate compared to benchmark schemes. The proposed policy is also shown to have faster training speed and higher communications efficiency with minimal loss of model accuracy.

Introduction

As mobile data traffic continues to surge, traditional cloud-centric architectures face significant challenges regarding backhaul congestion and high latency. Fog Radio Access Networks (F-RANs) address these issues by moving storage and computation to Fog Access Points (F-APs) closer to end-users. However, optimizing edge caching—deciding which content to store locally—is an NP-hard problem due to the dynamic nature of content popularity and limited local storage.

While Deep Reinforcement Learning (DRL) is effective for navigating such dynamic environments, training individual models at each F-AP is slow and data-intensive. Centralizing data for training poses privacy risks and consumes substantial bandwidth. Standard Federated Learning (FL) mitigates privacy concerns by sharing model weights instead of raw data, but it still suffers from "communication bloat" when transmitting large neural network updates over wireless links.

The FRLQ Framework

To overcome these challenges, the authors propose the Federated Reinforcement Learning with Quantization (FRLQ) framework. This approach integrates Dueling Deep Q-Networks (DQN) with a communication-efficient federated learning strategy.

Overview of the FRLQ framework integrating DRL and Federated Learning.

Dueling DQN for Local Decisions

Unlike standard DQNs, the Dueling DQN architecture splits the neural network into two streams: one to estimate the state-value function and another to estimate the "advantage" of specific actions. This separation allows the agent to learn which states are valuable without needing to learn the effect of every action for every state, resulting in more robust and faster-converging caching decisions.

Federated Learning and Quantization

FRLQ employs a three-pronged strategy to minimize communication overhead:

  1. Network Pruning: The system calculates "layer sensitivity" based on weight changes during training. Only the most sensitive layers are uploaded to the cloud server, skipping redundant updates.
  2. Quantization & Weight Sharing: K-means clustering is used to group similar weights. Instead of transmitting 32-bit floats for every connection, F-APs send small cluster indices, significantly compressing the model.
  3. Periodic Aggregation: Global model aggregation occurs only after several local iterations, reducing the frequency of communication rounds.
The FRLQ process: Weight quantization via K-means clustering and global model aggregation.

Experimental Evaluation

The proposed method was evaluated in a simulated F-RAN environment with 10 F-APs and a content library of 1,000 items. Content popularity followed a Mandelbrot-Zipf distribution.

Key Results

  • Cache Hit Rate: FRLQ significantly outperformed traditional heuristics like Least Recently Used (LRU) and Least Frequently Used (LFU). It also achieved faster performance gains than standard FL due to the reduced overhead of lighter models.
  • Communication Efficiency: By adjusting pruning thresholds, the authors reduced transmitted parameters to 30%–60% of the original model size with minimal impact on accuracy.
  • Latency Reduction: Average user request delay was lower than all benchmark schemes, particularly as content popularity became more skewed.

Conclusion

FRLQ provides a privacy-preserving and communication-efficient solution for edge caching in F-RANs. By combining Dueling DQN with aggressive model compression (pruning and quantization), it achieves high performance with significantly reduced network resource consumption. Future research will explore more complex topologies and asynchronous federated learning to further enhance scalability and responsiveness.

Summary updated

Abstract

In this paper, cooperative edge caching problem is studied in fog radio access networks (F-RANs). Given the non-deterministic polynomial hard (NP-hard) property of the problem, a dueling deep Q network (Dueling DQN) based caching update algorithm is proposed to make an optimal caching decision by learning the dynamic network environment. In order to protect user data privacy and solve the problem of slow convergence of the single deep reinforcement learning (DRL) model training, we propose a federated reinforcement learning method with quantization (FRLQ) to implement cooperative training of models from multiple fog access points (F-APs) in F-RANs. To address the excessive consumption of communications resources caused by model transmission, we prune and quantize the shared DRL models to reduce the number of model transfer parameters. The communications interval is increased and the communications rounds are reduced by periodical model global aggregation. We analyze the global convergence and computational complexity of our policy. Simulation results verify that our policy has better performance in reducing user request delay and improving cache hit rate compared to benchmark schemes. The proposed policy is also shown to have faster training speed and higher communications efficiency with minimal loss of model accuracy.
BibTeX
@misc{jiang2022federated,
  author = {Yanxiang Jiang and Min Zhang and Fu‐Chun Zheng and Yan Chen and Mehdi Bennis and Xiaohu You},
  title = {{A Federated Reinforcement Learning Method with Quantization for Cooperative Edge Caching in Fog Radio Access Networks}},
  year = {2022},
  eprint = {2206.11556},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.11556},
}
Figure