Structured Reinforcement Learning for Media Streaming at the Wireless Edge

Archana Bura, Sarat Chandra Bobbili, Shreyas Rameshkumar, Desik Rengarajan, Dileep Kalathil, Srinivas Shakkottai

Abstract

As video streaming continues to dominate mobile data traffic, the need for efficient resource allocation at the wireless edge has become critical. Traditional Deep Reinforcement Learning (DRL) approaches often struggle with the "curse of dimensionality" and high computational overhead, making them difficult to deploy in real-time network environments. This paper presents a structured reinforcement learning approach that exploits the underlying physics of queueing systems. By proving that the optimal resource allocation policy follows a threshold structure based on video buffer levels, we develop a lightweight algorithm that achieves a 30% improvement in Quality of Experience (QoE), trains four times faster than standard methods, and operates with an inference latency of just 15 microseconds.

Introduction

Video streaming currently accounts for approximately 70% of all downstream mobile data traffic. The transition toward "softwarized" wireless networks, driven by Software Defined Networking (SDN), offers a unique opportunity for intelligent controllers to manage radio resources dynamically. However, implementing intelligent control at the edge faces two primary hurdles: complexity and efficiency. In a typical WiFi or cellular cell with multiple clients, the state-action space grows exponentially, leading to the "curse of dimensionality." Furthermore, standard DRL models are computationally intensive and data-hungry, often failing to meet the microsecond-level decision requirements of high-speed network hardware.

This research addresses these challenges by moving away from "black-box" deep learning. Instead, we propose a structured approach that utilizes the mathematical properties of the streaming environment to create a more efficient and interpretable control policy.

Methodology

Problem Formulation

We formulate the resource allocation task as a Constrained Markov Decision Process (CMDP). The objective is to maximize the aggregate QoE across all connected clients while adhering to the physical capacity limits of the wireless access point.

Technical Innovations

To solve this CMDP efficiently, we introduce several key innovations:

  1. Lagrangian Relaxation: We decompose the complex centralized optimization problem into independent single-client sub-problems. A Lagrange multiplier serves as a virtual price signal, coordinating resource distribution among clients to ensure the total capacity constraint is respected.
  2. Threshold Policy Structure: We mathematically demonstrate that the optimal policy for an individual client follows a threshold structure. Specifically, the controller should prioritize a client if their video buffer falls below a specific value $X$, and deprioritize them otherwise.
  3. Structured Natural Policy Gradient (NPG): Rather than employing a deep neural network, we use an NPG algorithm to learn the parameters of a simple logistic function (a "single-neuron" model). This approach is mathematically guaranteed to converge to the global optimum while remaining computationally lightweight.
  4. Index-Based Heuristic: Inspired by the Whittle Index, we developed an "Index" version of the policy. By ranking clients based on their learned threshold values, the system becomes highly robust to fluctuations in channel conditions and the number of active users.

Experimental Setup

The proposed framework was evaluated using a dual-stage approach:

  • Simulation: A discrete-time simulator modeled on standard QoE metrics, such as the Delivery Quality Score (DQS), was used for initial training.
  • Real-World Testbed: We implemented a controller on a WiFi Access Point running OpenWRT and Linux Traffic Control (TC). The system managed live YouTube sessions on Intel NUC devices, totaling over 50 hours of streaming data. To stress-test the network, sessions were forced to 1080p resolution.

Key Results

The structured RL approach demonstrated significant advantages over both "vanilla" policies (e.g., standard WiFi round-robin) and unstructured DRL baselines:

  • QoE Enhancement: Average QoE increased by over 30%. Notably, 60-70% of sessions maintained a "perfect" QoE score of 5, significantly outperforming traditional methods which experienced frequent stalls.
  • Computational Efficiency: The decentralized training process was 4x faster than a centralized Proximal Policy Optimization (PPO) baseline.
  • Inference Latency: Decision-making was reduced to 10–15 microseconds, compared to the 50–60 microseconds required by unstructured deep RL models. This latency reduction is vital for integration into hardware-level controllers.
  • Robustness: The "Index" policy maintained high performance without retraining, even when the number of clients fluctuated or when the network experienced significant packet loss (up to 10%).

Discussion and Future Work

Our findings suggest that "structure beats scale" in the context of edge resource management. By exploiting the inherent threshold nature of video buffering, we can achieve superior performance with a fraction of the computational cost of deep learning.

However, limitations remain. The current model relies on training within a simulator; future iterations could benefit from fine-tuning on live traffic to better accommodate non-YouTube streaming services. Additionally, while the system is robust to static interference, further research is required to evaluate performance in high-mobility scenarios where rapid physical occlusions may affect threshold stability. Expanding the QoE model to account for diverse streaming protocols will also be a priority for future development.

Summary updated

BibTeX
@inproceedings{bura2024structured,
  author = {Archana Bura and Sarat Chandra Bobbili and Shreyas Rameshkumar and Desik Rengarajan and Dileep Kalathil and Srinivas Shakkottai},
  title = {{Structured Reinforcement Learning for Media Streaming at the Wireless Edge}},
  booktitle = {MobiHoc},
  year = {2024},
  doi = {10.1145/3641512.3686386},
}
Figure