Deep Reinforcement Learning-Based Block Coordinate Descent for Downlink Weighted Sum-Rate Maximization on AI-Native Wireless Networks

Siya Chen, Chee Wei Tan, H. Vincent Poor

TL;DR

This paper introduces a novel Deep Reinforcement Learning-based Block Coordinate Descent (DRL-based BCD) algorithm to efficiently solve the complex problem of maximizing weighted sum-rate in wireless networks. By combining the data-driven learning of DRL with the optimization insights of BCD, the proposed method achieves higher accuracy and robustness, potentially reaching optimal solutions while being computationally efficient.

What Problem Does This Paper Address?

Modern wireless networks are becoming increasingly complex, demanding more intelligent and adaptive resource allocation strategies. A key challenge is maximizing the total data throughput, known as the weighted sum-rate maximization (WSRM) problem, especially under limited transmission power. This problem is notoriously difficult because it's non-convex, making it hard for traditional optimization methods to find globally optimal solutions efficiently. Existing deep learning approaches often struggle to capture the problem's inherent structure, leading to suboptimal performance and requiring extensive tuning. This paper aims to bridge this gap by integrating the strengths of both deep reinforcement learning and optimization theory.

Methods

The core innovation is the DRL-based BCD algorithm. It builds upon a standard Block Coordinate Descent (BCD) method, which iteratively optimizes one block of variables at a time. While BCD is efficient and leverages theoretical properties of the problem, it can get stuck in local optima. To overcome this, the authors integrate Deep Reinforcement Learning (DRL), specifically an Actor-Critic framework.

Here's how it works:

  • State Representation: The state s(t) for the DRL agent is a rich combination of system parameters (channel gains, noise, power), the current data rate γ(t) obtained from the BCD iteration, and features derived from the problem's structure. This allows the DRL agent to learn from both the environment's current state and the underlying optimization problem's characteristics.
  • Action Space: Instead of directly controlling power allocation (which can be challenging), the DRL agent learns to predict favorable starting points for the BCD's iterative updates of the γ variable. This is a key insight, as it leverages the fast convergence of the γ updates within BCD.
  • Reward Function: The agent is rewarded for quickly converging to the optimal sum-rate. This encourages efficient exploration and faster convergence.
  • Integration: The DRL agent's actions guide the BCD's subproblem solving, and the BCD's progress (data rate γ) informs the DRL agent's state. This creates a synergistic loop where DRL helps BCD escape local optima, and BCD provides structured information for DRL.

The paper also explores two specific DRL implementations: DDPG-based BCD and TD3-based BCD, which are standard deep reinforcement learning algorithms adapted for this problem.

Datasets

The paper does not explicitly mention pre-existing public datasets. Instead, it describes the generation of synthetic WSRM problem instances for training and evaluation. These instances are created with varying numbers of users (L), antennas at the base station (N), and receiver antennas (Nl), allowing for the assessment of the algorithm's scalability and performance across different network configurations. For smaller scenarios (L=4), optimal solutions were obtained via brute-force search or established algorithms to serve as ground truth.

Key Insights and Ideas

  • DRL for Optimization Initialization: A crucial insight is using DRL not to solve the entire optimization problem directly, but to intelligently guide the initialization and iterative steps of a traditional optimization algorithm (BCD). This leverages DRL's exploration capabilities to overcome the local optima problem inherent in BCD.
  • Action on γ vs. Power: Directing the DRL agent to learn actions that influence the γ variable (related to signal-to-interference-plus-noise ratio) rather than directly on power allocation is shown to be more effective. This is because γ updates within BCD are fast and have a more direct link to the objective function.
  • State Representation Matters: Including structural features of the WSRM problem (like Fp(γ) + σ) in the DRL state significantly enhances performance, allowing the agent to learn from the problem's mathematical properties, not just raw data.
  • Extensibility: The DRL-based BCD framework is demonstrated to be highly extensible, successfully applied to more complex scenarios like joint beamforming, showcasing its broad applicability.

Key Results

  • Improved Accuracy and Robustness: The DRL-based BCD models (DDPG-based and TD3-based) consistently achieve higher sum-rate accuracy and exhibit lower sum-rate errors compared to purely DRL baselines (DDPG, TD3, GNN) and the standard BCD method, especially across various network scales.
  • Faster Convergence (for DRL-based BCD): While pure BCD might converge in fewer steps, the DRL-based BCD models achieve better sum-rate accuracy with comparable or faster convergence of the actor network compared to purely DRL models.
  • Reduced Training Time: The DRL-based BCD models generally require less training time than purely DRL models, likely due to the guidance provided by the BCD structure.
  • Scalability: The algorithm demonstrates good performance across different network sizes (L=4 to L=128 users).
  • Beamforming Extension: The framework is successfully extended to joint beamforming, showing its potential for more complex wireless optimization tasks.

Limitations and Future Work

  • Computational Cost: While the DRL-based BCD models are efficient, the environment updates within the DRL framework can add computational overhead, especially in scenarios with many transmitter antennas. The paper suggests that for larger antenna counts, simpler BCD with suboptimal precoders might be sufficient.
  • Hyperparameter Sensitivity: Although the DRL-based BCD models are shown to be more robust to hyperparameter variations than purely DRL models, careful tuning is still required.
  • Future Work: The authors suggest exploring distributed algorithms with lower computational complexity and investigating new optimization methods for other complex non-convex problems in AI-native wireless networks. They also plan to further integrate mathematical structures with data-driven techniques.

Summary updated

Abstract

This paper introduces a deep reinforcement learning-based block coordinate descent (DRL-based BCD) algorithm to address the nonconvex weighted sum-rate maximization (WSRM) problem with a total power constraint. Firstly, we present an efficient block coordinate descent (BCD) method to solve the problem. We then integrate deep reinforcement learning (DRL) techniques into the BCD method and propose the DRL-based BCD algorithm. This approach combines the data-driven learning capability of machine learning techniques with the navigational and decision-making characteristics of the optimization-theoretic-based BCD method. This combination significantly improves the algorithm's performance by reducing its sensitivity to initial points and mitigating the risk of entrapment in local optima. The primary advantages of the proposed DRL-based BCD algorithm lie in its ability to adhere to the constraints of the WSRM problem and significantly enhance accuracy, potentially achieving the exact optimal solution. Moreover, unlike many pure machine-learning approaches, the DRL-based BCD algorithm capitalizes on the underlying theoretical analysis of the WSRM problem's structure. This enables it to be easily trained and computationally efficient while maintaining a level of interpretability. Through numerical experiments, the DRL-based BCD algorithm demonstrates substantial advantages in effectiveness, efficiency, robustness, and interpretability for maximizing sum rates, which also provides valuable potential for designing resource-constrained AI-native wireless optimization strategies in next-generation wireless networks.
BibTeX
@article{chen2026deep,
  author = {Siya Chen and Chee Wei Tan and H. Vincent Poor},
  title = {{Deep Reinforcement Learning-Based Block Coordinate Descent for Downlink Weighted Sum-Rate Maximization on AI-Native Wireless Networks}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2026},
  doi = {10.1109/twc.2025.3605329},
}
Figure