Abstract
Managing interference in heterogeneous networks (HetNets) is critical for maximizing system throughput. Traditional centralized power control methods suffer from high signaling overhead and poor scalability. This paper proposes a novel Penalty-based Q-learning (PQL) algorithm within a Distributed-Training-and-Execution (DTE) framework. By introducing a regularization term to the loss function, PQL stabilizes the multi-agent learning environment, allowing access points (APs) to independently learn cooperative power allocation policies. Simulation results demonstrate that PQL achieves over 90% of the performance of centralized benchmarks with minimal local information exchange.
Introduction
The deployment of small cells (femtocells, picocells) within macrocell coverage—forming a HetNet—is a key strategy for meeting increasing data demands. However, the resulting cross-tier and co-tier interference necessitates precise power control. Existing solutions often rely on Centralized Training and Decentralized Execution (CTDE), which still faces scalability issues during the training phase. This work shifts towards a fully distributed approach where both training and execution are decentralized, reducing the reliance on global channel state information (CSI).
System Model and Problem Formulation
In a HetNet, multiple APs serve their respective user devices (UDs) simultaneously over the same frequency band. The primary challenge is that each AP's transmission acts as interference for UDs served by other APs.
As illustrated in the figure above, the channel gain between AP $j$ and UD $i$ is denoted as $h_{i,j}$. The goal is to optimize the transmit power of each AP to maximize the sum-throughput of the network while operating under local constraints. The environment is highly dynamic, with path loss following the 3GPP TR 25.942 standard and small-scale fading modeled by the Jakes fading model.
Methodology: Penalty-based Q-learning (PQL)
The core of the proposed solution is the PQL algorithm. Multi-agent reinforcement learning often suffers from non-stationarity: as one agent updates its policy, the environment perceived by others changes, creating a "moving target" problem. PQL addresses this through three key mechanisms:
- Loss Function Regularization: When an agent identifies a high-reward action, a penalty is added to the loss function for all other unselected actions. This effectively "widens the gap" between the best action and sub-optimal ones.
- Policy Stabilization: By artificially making the "best" action stand out more clearly and for longer, the agent's policy becomes more predictable. This "stubbornness" provides a stable environment for neighboring agents to adapt and learn how to cooperate.
- Dueling DQN Architecture: Each agent employs a Dueling Deep Q-Network to separately estimate state values and action advantages, further stabilizing the learning process.
To drive global optimization through local actions, each agent's reward is defined as the average achievable rate of itself and its $M$ nearest neighbors. This local "altruism" requires only the exchange of two scalar values (achievable rate and interference power) with neighbors, making it highly practical for real-world backhaul constraints.
Performance Evaluation
The algorithm was evaluated in a simulated three-tier HetNet with 9 to 13 APs. The action space was discretized into 11 power levels. Key findings include:
- Near-Optimal Performance: PQL reaches over 90% of the sum-throughput of the WMMSE algorithm, a classic centralized benchmark that requires perfect global CSI.
- Superiority over Baselines: PQL consistently outperformed Independent Q-Learning (IQL) and Hysteretic Q-Learning (HQL) by approximately 3-5%.
- Scalability and Robustness: The performance gap between PQL and other MADRL methods widened as the network became denser. Furthermore, PQL remained stable even when the temporal correlation of the channel was zero, proving its ability to handle highly dynamic environments.
Conclusion
The proposed PQL framework enables effective distributed power control in HetNets. By prioritizing policy stability through a novel penalty mechanism, it overcomes the limitations of centralized and CTDE-based methods, offering a scalable and high-performing solution for modern wireless infrastructure. Future work may explore continuous action spaces and the impact of backhaul latency on convergence.
Summary updated