Abstract
The integration of caching and ultra-dense network (UDN) can not only improve the efficiency of content retrieval by reducing duplicate content transmissions but also improve the network throughput and system energy efficiency (EE) of the UDN. In this paper, we focus on energy consumption aspects of cache-aided UDN (CUDN) and develop a novel caching strategy to improve the system EE. Different from the existing researches, we consider a more realistic scenario where the popularity of the cache content is dynamic and unknown. The proposed caching strategy is a deep reinforcement learning (DRL)-based approach that uses a deep Q-network to approximate the Q action-value function. We optimize the structure and corresponding parameters of the deep Q-network according to the latest findings in the field of DRL and deep learning (DL) to improve the performance of this caching strategy. The simulation results show that the performance in terms of EE of the CUDN can be significantly improved by using our proposed caching strategy.
Introduction
As we move toward 5G and beyond, Ultra-Dense Networks (UDNs)—which pack numerous small access points (SAPs) into a small area—are essential for meeting high data demands. However, these networks face massive backhaul pressure and high energy consumption. Caching popular content at the "edge" (directly on SAPs) is a proven solution to reduce redundant transmissions, but it introduces a complex optimization problem: What should be cached?
Most existing research assumes that content popularity is either fixed or perfectly known. In the real world, what users want changes constantly (dynamic) and is rarely known in advance (unknown). This paper tackles the challenge of maximizing Energy Efficiency (EE) under these unpredictable, non-stationary conditions.
Methods: The ADRLC Strategy
The authors frame the caching problem as a Markov Decision Process (MDP) and solve it using a Deep Q-Network (DQN) approach. Their primary contribution is the Advanced DRL-based Cache (ADRLC) strategy, which improves upon standard DRL through three key architectural choices:
- Recurrent Neural Networks (RNNs): Unlike standard deep networks, RNNs are designed to handle sequences. Since content popularity evolves over time, the RNN allows the agent to extract temporal correlations from request patterns.
- Prioritized Experience Replay (PER): Instead of learning from past experiences at random, the agent prioritizes "surprising" or highly informative transitions (those with a high temporal-difference error), leading to faster and more stable learning.
- Dueling Architecture: This splits the Q-function into two parts: one that estimates the value of being in a certain state and another that estimates the advantage of a specific action. This helps the model better evaluate which caching decisions are truly beneficial when many options have similar values.
The reward function for the agent is the system's Energy Efficiency (EE), defined as the ratio of total network throughput to total power consumption (including transmit, operational, and data retrieval power).
Performance Evaluation
The researchers evaluated their strategy using both synthetic and real-world data:
- Synthetic Distributions: They used Uniform and Zipf distributions to simulate stationary popularity environments.
- MovieLens Dataset: To test dynamic, real-world performance, they used the MovieLens 1M dataset (1 million movie ratings). They treated ratings as content requests and used the timestamps to simulate a time-sensitive, non-stationary popularity environment.
Key Results
- Faster Convergence: In simulations using the MovieLens dataset, the ADRLC algorithm reached maximum energy efficiency in significantly fewer learning steps than standard DRL or existing CNN-based methods.
- Superior Efficiency: ADRLC consistently achieved higher converged EE compared to random caching and standard DRL baselines. In some scenarios, it improved energy efficiency by 40% to 75% over existing benchmarks.
- Scalability and Robustness: The strategy remained effective across varying access point densities and different power settings for local vs. backhaul data retrieval.
Conclusion
The ADRLC algorithm demonstrates that treating content popularity as a correlated sequence via RNNs significantly enhances caching performance in ultra-dense environments. While the study uses a simplified power model, the results highlight a clear path toward more energy-efficient "green" networks. Future work will focus on dynamic network topologies and managing the computational complexity associated with very large content libraries.
Summary updated