Abstract
Open Radio Access Network (O-RAN) is an emerging paradigm proposed for enhancing the 5G network infrastructure. O-RAN promotes open vendor-neutral interfaces and virtualized network functions that enable the decoupling of network components and their optimization through intelligent controllers. The decomposition of base station functions enables better resource usage, but also opens new technical challenges concerning their efficient orchestration and allocation. In this paper, we propose Proactive Resource Orchestrator based on Reinforcement Learning (PRORL), a novel solution for the efficient and dynamic allocation of resources in O-RAN infrastructures. We frame the problem as a Markov Decision Process and solve it using Deep Reinforcement Learning; one relevant feature of PRORL is that it learns demand patterns from experience for proactive resource allocation. We extensively evaluate our proposal by using both synthetic and real-world data, showing that we can significantly outperform the existing algorithms, which are typically based on the analysis of static demands. More specifically, we achieve an improvement of 90% over greedy baselines and deal with complex trade-offs in terms of competing objectives such as demand satisfaction, resource utilization, and the inherent cost associated with allocating resources.
Introduction and Problem Statement
The transition toward 5G and Open RAN (O-RAN) architectures has introduced a "disaggregated" model where traditional base station functions are split into Central Units (CUs), Distributed Units (DUs), and Radio Units (RUs). While this virtualization offers immense flexibility, it creates a complex orchestration challenge: how to efficiently allocate finite compute resources across different network nodes in real-time.
As shown in Figure 1, the O-RAN architecture spans multiple tiers, from Regional Clouds to Edge Data Centers and individual Cell Sites. Static over-provisioning—setting resources to meet peak demand—is economically wasteful, while reactive allocation often fails to keep pace with sudden traffic spikes. Furthermore, moving virtualized resources between "Points of Presence" (PoPs) incurs operational expenditure (OPEX) costs. PRORL addresses these challenges by providing a solution that is both proactive (anticipating demand) and multi-objective (balancing service quality, resource utilization, and movement costs).
The PRORL Framework
PRORL frames resource orchestration as a Multi-Objective Markov Decision Process (MOMDP), solved using an advanced Deep Reinforcement Learning (DRL) architecture. The system interacts with the network environment to learn optimal policies for resource movement.
Key Innovations
- Action Space Decomposition: To manage the exponential growth of possible allocation combinations, PRORL splits the agent into two sequential sub-agents: an "Add Sub-Agent" and a "Remove Sub-Agent." This linearization significantly accelerates convergence and improves policy quality.
- Advanced DRL Architecture: The system utilizes Double Dueling Deep Q-Networks (DQN) with Prioritized Experience Replay (PER). This combination prevents overestimation bias and focuses learning on the most critical network transitions.
- Multi-Objective Utility: The agent optimizes a scalarized reward function weighting three key metrics: Surplus (minimizing idle resources), Cost (minimizing OPEX from resource movements), and Gap (minimizing the shortfall between demand and capacity to ensure QoS).
Evaluation and Results
The framework was evaluated using both real-world Call Detail Records (CDRs) from Milan and synthetic datasets designed to test stress scenarios like weekly commuter cycles and global overdemand.
PRORL demonstrated a 90% performance improvement over traditional greedy baselines across various trade-off configurations. In synthetic stress tests, the model achieved up to 10x performance gains in specific objectives, such as minimizing movement costs. Notably, in "Global Overdemand" scenarios, PRORL intelligently limited resource movements to save costs when it recognized that total demand satisfaction was mathematically impossible, whereas baseline algorithms continued to fruitlessly "thrash" resources.
Conclusion
PRORL represents a significant step forward in O-RAN orchestration, offering a proactive, efficient, and tunable solution for managing virtualized resources. With an execution time of less than 2 milliseconds per decision, it is well-suited for near-real-time O-RAN controllers. Future work will focus on integrating PRORL into lower levels of the O-RAN stack and testing within large-scale live testbeds.
Summary updated