Abstract
Load balancing (LB) is a critical function in cellular networks, aimed at redistributing traffic to optimize Quality of Service (QoS). Traditional Reinforcement Learning (RL) approaches to LB are often hindered by the difficulty of manual reward engineering. This paper presents a framework that utilizes Inverse Reinforcement Learning (IRL) to automatically derive reward functions from ranked demonstrations. By introducing Temporally Consistent Sampling (TCS), the proposed method accounts for the periodic nature of network traffic, enabling the agent to outperform the sub-optimal experts from which it learns. Experimental results in a system-level simulator demonstrate significant improvements in throughput, fairness, and congestion management compared to existing rule-based and standard RL baselines.
Introduction
In modern cellular networks, traffic distribution is frequently imbalanced; it is common for over 50% of total traffic to be concentrated in as few as 15% of cells. This imbalance leads to localized congestion, increased latency, and a degradation of the overall user experience. Load Balancing (LB) techniques seek to mitigate these issues by shifting users from heavily loaded cells to those with available capacity.
While Reinforcement Learning (RL) has shown promise in automating LB, its success is heavily dependent on the design of a reward function. In the context of telecommunications, there is no industry-wide consensus on how to weight competing Key Performance Indicators (KPIs) such as minimum throughput, average fairness, and congestion levels. This "reward engineering" bottleneck often results in brittle policies that fail to generalize across different network conditions.
Methodology
Learning from Ranked Demonstrations
To bypass the need for manual reward design, this research employs a variation of Inverse Reinforcement Learning known as Trajectory Extrapolation (T-REX). Unlike traditional IRL, which requires expert demonstrations, T-REX learns from a set of ranked sub-optimal trajectories. By observing that one sequence of actions resulted in a better outcome than another, the reward network can infer the underlying objective and extrapolate a policy that exceeds the performance of the original demonstrations.
Temporally Consistent Sampling (TCS)
The primary technical innovation of this work is Temporally Consistent Sampling (TCS). Network traffic is inherently periodic, following distinct diurnal patterns (e.g., high usage during the day, low usage at night). Standard T-REX implementations sample random snippets of data for comparison. However, comparing a high-performance snippet from a low-traffic period (3 AM) to a lower-performance snippet from a peak-traffic period (2 PM) is misleading; the difference in performance is likely due to external conditions rather than the agent's actions.
TCS addresses this by ensuring that when the algorithm compares two trajectories, it samples the exact same time indices from both. This temporal alignment allows the reward network to isolate the impact of the agent’s load-balancing decisions from the background noise of periodic traffic fluctuations. The researchers found that TCS reduced "label noise"—instances where the system misidentifies which action was superior—from 35% to 15%.
Training Pipeline
The final implementation follows a three-step process:
- Data Collection: 100 trajectories are generated using a random controller and ranked based on a high-level KPI aggregate.
- Reward Learning: A Reward Network is trained using the ranked data and the TCS mechanism.
- Policy Optimization: The learned reward function is used to train a final policy via Proximal Policy Optimization (PPO).
Experimental Setup
The framework was evaluated using a system-level Radio Access Network (RAN) simulator. The environment consisted of seven base stations (eNBs) arranged in a hexagonal grid. The model was tested across four distinct traffic scenarios, varying in user density, packet sizes, and arrival intervals to test the robustness of the learned reward function.
Results and Discussion
The IRL + TCS method was compared against fixed rule-based systems, adaptive rules, and standard T-REX. The proposed method demonstrated superior performance across all primary metrics:
- Minimum Throughput ($T_{min}$): Achieved a 19.6% average improvement over the strongest baseline, ensuring better service for the worst-off users.
- Fairness ($T_{std}$): Reduced the standard deviation of throughput by 26.7%, indicating a more equitable distribution of network resources.
- Congestion ($T_{cc}$): Reduced the number of congested cells by 32.3%.
Furthermore, the inclusion of TCS significantly improved the Pearson correlation between the predicted reward and actual network performance. This was particularly evident in "extrapolation" tasks, where the model had to predict rewards for network states that were superior to any seen during the initial demonstration phase.
Conclusion
By shifting the focus from manual reward engineering to automated reward inference, this framework provides a more scalable and robust solution for communication load balancing. The introduction of Temporally Consistent Sampling is a vital contribution for applying IRL to domains with strong external cycles, such as telecommunications and power grids.
Future work will focus on improving the data efficiency of the IRL process and exploring the scalability of this approach in massive, multi-vendor heterogeneous networks. Additionally, the authors suggest that this framework could be adapted for other network optimization tasks, including energy saving and network slicing.
Summary updated