Structured Reinforcement Learning for Delay-Optimal Data Transmission in Dense mmWave Networks

Shufan Wang, Guojun Xiong, Shichen Zhang, Huacheng Zeng, Jian Li, Shivendra S. Panwar

Abstract

We study the data packet transmission problem (mmDPT) in dense cell-free millimeter wave (mmWave) networks, where users send data packet requests to access points (APs) via uplinks and APs transmit requested data packets to users via downlinks. Our objective is to minimize the average delay in the system due to APs' limited service capacity and unreliable wireless channels. This problem is formulated as a restless multi-armed bandits problem with fairness constraint (RMAB-F). Since finding the optimal policy for RMAB-F is intractable, we propose a structured reinforcement learning (RL) solution called mmDPT-TS. By exploiting the inherent structure of the index policy, mmDPT-TS achieves a near-optimal Bayesian regret of $\tilde{O}(\sqrt{T})$ while remaining computationally efficient for practical implementation. Extensive emulation based on realistic mmWave network data demonstrates significant gains over existing approaches.

Introduction: The Challenge of Dense mmWave Networks

Millimeter-wave (mmWave) technology is a cornerstone of next-generation wireless communication, offering the massive bandwidth required for high-speed data transmission. However, mmWave signals are highly susceptible to blockages and mobility, leading to intermittent connectivity. To mitigate this, dense cell-free networks deploy numerous Access Points (APs) to serve a population of users, ensuring that at least one line-of-sight path is likely available.

System model of a dense mmWave network where multiple APs serve a population of users under unreliable channel conditions.

The core challenge in these networks is the Data Packet Transmission (mmDPT) problem: determining which AP should handle a user's request to minimize delay. This decision is complicated by time-varying downlink quality, limited AP service capacity, and the need for fairness among users. In dense networks, the state space—defined by queue lengths across all AP-user pairs—grows exponentially, rendering standard RL algorithms like DQN or standard Thompson Sampling computationally prohibitive.

Methodology: Structured Reinforcement Learning

To overcome the "curse of dimensionality," this research proposes a structured RL approach that leverages the mathematical properties of the network model.

RMAB-F Formulation

The system is modeled as a Restless Multi-Armed Bandit with Fairness (RMAB-F). Unlike traditional bandits, "restless" arms (user-AP pairs) change state (queue length) even when they are not actively selected. The fairness constraint ensures that resources are distributed equitably across the user base.

The mmDPT-TS Algorithm

Instead of treating the network as a black box, the authors develop mmDPT-TS, which combines Thompson Sampling with a low-complexity index policy.

  1. Learning: The algorithm uses Thompson Sampling to maintain and update beliefs about unknown transition probabilities (channel qualities).
  2. Decision Making: Rather than solving complex Bellman equations, it calculates a "value index" for each user-AP pair using Linear Programming (LP) based on occupancy measures. This index represents the immediate benefit of assigning a request to a specific AP.
  3. Scheduling: Requests are sorted by their indices and assigned greedily to APs until capacity limits are reached, naturally satisfying the fairness constraints.

Experimental Setup and Realistic Traces

The algorithm was evaluated using a combination of synthetic traces and real-world data collected from a custom 60GHz mmWave testbed. The hardware setup utilized Software Defined Radios (USRP X310) and ADI 60GHz RF frontends to capture the nuances of indoor signal propagation.

EVM traces collected from a 60GHz testbed, illustrating the high volatility of mmWave channels over time.

Researchers collected over 460,000 Error Vector Magnitude (EVM) samples in a conference room environment. These traces, shown in the figure above, highlight the rapid fluctuations in channel quality that the mmDPT-TS algorithm must navigate in real-time.

Key Results and Performance

The proposed mmDPT-TS algorithm demonstrates several critical advantages:

  • Theoretical Efficiency: It achieves a Bayesian regret of $O(\sqrt{T} \log T)$, the theoretical gold standard for online learning efficiency, ensuring that performance converges rapidly to the optimal policy.
  • Delay Reduction: In both testbed-driven emulations and synthetic scalability tests (up to 100 users), mmDPT-TS significantly outperformed Whittle Indexing, DQN, and Soft Actor-Critic (SAC) in minimizing accumulated system delay.
  • Computational Speed: By avoiding iterative value iteration and focusing on index calculation, mmDPT-TS is significantly faster than traditional index-based methods, making it suitable for high-frequency 5G/6G hardware.
  • Asymptotic Optimality: Numerical results confirm that as the network density increases, the performance gap between mmDPT-TS and the theoretical optimal policy effectively closes to zero.

Conclusion and Future Work

By exploiting the inherent structure of the RMAB-F problem, mmDPT-TS provides a scalable and efficient solution for data transmission in dense mmWave networks. While currently designed for centralized control, future research may explore decentralized or federated versions to reduce signaling overhead in massive-scale deployments. Additionally, incorporating explicit models for high-speed mobility remains a promising direction for extending this framework to vehicular and industrial IoT environments.

Summary updated

Abstract

We study the data packet transmission problem (mmDPT) in dense cell-free millimeter wave (mmWave) networks, i.e., users sending data packet requests to access points (APs) via uplinks and APs transmitting requested data packets to users via downlinks. Our objective is to minimize the average delay in the system due to APs' limited service capacity and unreliable wireless channels between APs and users. This problem can be formulated as a restless multi-armed bandits problem with fairness constraint (RMAB-F). Since finding the optimal policy for RMAB-F is intractable, existing learning algorithms are computationally expensive and not suitable for practical dynamic dense mmWave networks. In this paper, we propose a structured reinforcement learning (RL) solution for mmDPT by exploiting the inherent structure encoded in RMAB-F. To achieve this, we first design a low-complexity and provably asymptotically optimal index policy for RMAB-F. Then, we leverage this structure information to develop a structured RL algorithm called mmDPT-TS, which provably achieves an \tilde{O}(\sqrt{T}) Bayesian regret. More importantly, mmDPT-TS is computation-efficient and thus amenable to practical implementation, as it fully exploits the structure of index policy for making decisions. Extensive emulation based on data collected in realistic mmWave networks demonstrate significant gains of mmDPT-TS over existing approaches.
BibTeX
@article{wang2024structured,
  author = {Shufan Wang and Guojun Xiong and Shichen Zhang and Huacheng Zeng and Jian Li and Shivendra S. Panwar},
  title = {{Structured Reinforcement Learning for Delay-Optimal Data Transmission in Dense mmWave Networks}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2024},
  url = {https://openreview.net/forum?id=0DoxXBFo7g},
}
Figure