Collaborative Edge Caching: a Meta Reinforcement Learning Approach with Edge Sampling

Yinan Mao, Bowei He, Shiji Zhou, Chen Ma, Zhi Wang

Abstract

Edge caching is a critical technology for reducing network latency and backhaul traffic by storing popular content at the network periphery. However, existing caching strategies often struggle with the highly dynamic nature of content popularity and the data sparsity inherent in localized edge servers. This paper presents CMCES (Collaborative Meta Reinforcement Learning Caching with Edge Sampling), a framework designed to optimize cache hit rates through fast adaptation and selective collaboration. By integrating Meta-Reinforcement Learning (Meta-RL) with an intelligent edge sampling mechanism, CMCES effectively leverages neighbor server data while mitigating the risks of "negative transfer" from heterogeneous request distributions. Experimental results on large-scale real-world traces demonstrate that CMCES achieves up to a 10.12% improvement in cache performance over state-of-the-art methods with minimal communication overhead.

Introduction

As mobile data traffic continues to surge, edge caching has emerged as a vital solution to bring content—such as viral videos and social media updates—closer to the end-user. By storing frequently accessed data on local servers, providers can significantly reduce latency and alleviate network congestion. Despite its benefits, edge caching faces two primary challenges:

  1. Dynamic Popularity: Content popularity is highly volatile. Traditional heuristic algorithms like Least Recently Used (LRU) or Least Frequently Used (LFU) are often too reactive, failing to anticipate rapid shifts in user interest.
  2. Data Sparsity and Heterogeneity: Individual edge servers often lack sufficient request data to train robust Deep Reinforcement Learning (DRL) models. While collaborative caching allows servers to share data, neighboring nodes often serve different demographics (e.g., residential vs. business districts). Blindly merging this heterogeneous data can lead to "negative transfer," where irrelevant information degrades the local model's performance.

The CMCES Framework

To address these challenges, the CMCES framework introduces a two-stage approach that combines the fast adaptation of meta-learning with a selective communication strategy.

Collaborative Meta-Reinforcement Learning

Unlike standard DRL, which trains a model for a specific state, Meta-RL focuses on "learning how to learn." CMCES utilizes Meta-RL to find a meta-initialization—a set of model weights that can be rapidly adapted to new local content distributions with minimal gradient steps. The system creates "task pairs" from consecutive time intervals, treating the evolving popularity of content as a series of related tasks.

Furthermore, the framework incorporates Neighbor-Local Task Pairs. In this setup, the meta-loss function is calculated using historical data from a neighboring server to predict future requests at the local node. This process trains the model to identify and extract useful features from its peers, effectively bridging the data gap for servers with low traffic.

Adaptive Edge Sampling

To solve the heterogeneity problem, CMCES implements an Edge Sampling Strategy. Rather than treating all neighbors equally, each local node maintains a combination matrix that dynamically adjusts the weight of incoming data. Neighbors with similar request patterns are assigned higher weights, while those with dissimilar distributions are sampled less frequently. This "selective listening" acts as a filter, ensuring that only relevant information influences the local caching policy.

Decentralized and Efficient Execution

CMCES is designed for practical deployment in bandwidth-constrained environments. The framework utilizes decentralized execution, where nodes exchange small model updates rather than raw datasets. Each exchange is approximately 0.9KB, resulting in a 31% reduction in communication costs compared to traditional collaborative DRL methods.

Experimental Evaluation

The researchers evaluated CMCES using two massive, real-world request traces from Chinese platforms:

  • IQIYI: A long-video platform trace containing over 53 million requests for 417,000 unique contents.
  • KuaiShou: A short-video platform trace (similar to TikTok) containing over 528 million requests for 1.7 million contents. This dataset is characterized by extreme volatility, providing a rigorous test for adaptation speed.

The simulation utilized 56 representative edge areas in Beijing to model a realistic geographic distribution of servers.

Key Results and Insights

  • Superior Hit Rates: CMCES consistently outperformed baselines, including Multi-agent DRL (MADRL) and traditional rules. It achieved a 9.3% improvement on IQIYI and a 1.7% improvement on KuaiShou (representing a ~10% normalized improvement given the difficulty of the dataset).
  • Resilience to Volatility: The performance gains were most pronounced on the KuaiShou dataset. The fast-adaptation capability of Meta-RL allowed the system to "ride the trend" of short-video popularity more effectively than static or slowly-adapting models.
  • Scalability: As the number of neighboring devices increased from 1 to 8, the hit rate of CMCES continued to improve. This demonstrates that the edge sampling mechanism successfully extracts value from larger pools of neighbors without being overwhelmed by data heterogeneity.
  • Communication Efficiency: By maintaining a package size of only 0.9KB per exchange, the model proves highly viable for real-world edge networks where backhaul capacity is a premium.

Limitations and Future Outlook

While CMCES represents a significant step forward, several areas for future research remain. The current study assumes a fixed topology; however, in scenarios like vehicular edge computing, the neighbor list is highly dynamic, requiring a more fluid sampling matrix. Additionally, the reward function is currently limited to hit rates; future iterations could incorporate energy costs and varying content sizes. Finally, integrating security measures against potential poisoning attacks from malicious neighbors would enhance the framework's robustness for public infrastructure deployment.

Conclusion

CMCES addresses the dual hurdles of dynamic popularity and data heterogeneity in edge caching. By combining the adaptive power of Meta-RL with a selective edge sampling strategy, the framework provides a scalable, efficient, and high-performance solution for modern content delivery networks.

Summary updated

BibTeX
@inproceedings{mao2023collaborative,
  author = {Yinan Mao and Bowei He and Shiji Zhou and Chen Ma and Zhi Wang},
  title = {{Collaborative Edge Caching: a Meta Reinforcement Learning Approach with Edge Sampling}},
  booktitle = {Proceedings - IEEE International Conference on Multimedia and Expo},
  year = {2023},
  url = {https://openreview.net/forum?id=HU749hQ09U},
}
Figure