Self-play Reinforcement Learning for Video Transmission

Tianchi Huang, Rui-Xiao Zhang, Lifeng Sun

Abstract

Video transmission services adopt adaptive algorithms to ensure users' demands. Existing techniques are often optimized and evaluated by a function that linearly combines several weighted metrics. Nevertheless, we observe that the given function fails to describe the requirement accurately. Thus, such proposed methods might eventually violate the original needs. To eliminate this concern, we propose Zwei, a self-play reinforcement learning algorithm for video transmission tasks. Zwei aims to update the policy by straightforwardly utilizing the actual requirement. Technically, Zwei samples a number of trajectories from the same starting point and instantly estimates the win rate w.r.t the competition outcome. Here the competition result represents which trajectory is closer to the assigned requirement. Subsequently, Zwei optimizes the strategy by maximizing the win rate. To build Zwei, we develop simulation environments, design adequate neural network models, and invent training methods for dealing with different requirements on various video transmission scenarios. Trace-driven analysis over two representative tasks demonstrates that Zwei optimizes itself according to the assigned requirement faithfully, outperforming the state-of-the-art methods under all considered scenarios.

Introduction

In modern video streaming, maintaining a high Quality of Experience (QoE) is a complex balancing act. Traditional Adaptive Bitrate (ABR) algorithms rely on Deep Reinforcement Learning (DRL) trained via hand-tuned reward functions. These functions typically combine metrics like bitrate and rebuffering time into a single scalar value using linear weights. However, this approach often fails because fixed weights cannot capture the deterministic priorities of human users across diverse network conditions (e.g., a stable fiber connection versus a fluctuating mobile signal). When the reward function is misaligned with actual user requirements, the AI learns sub-optimal strategies that may technically maximize a score while providing a poor viewing experience.

Methodology: The Zwei Framework

To overcome the limitations of reward engineering, we introduce Zwei, a framework that utilizes self-play to optimize policies directly against user requirements. Instead of chasing a numerical reward, Zwei learns through comparison and competition.

Self-Play and Trajectory Sampling

Zwei employs a Monte Carlo sampling approach to generate multiple potential "trajectories" (sequences of bitrate decisions) from a single starting point in a network trace. These trajectories are then compared in pairs using a deterministic "Rule" that reflects actual user priorities (e.g., "minimize rebuffering first, then maximize bitrate").

Zwei utilizes self-play by sampling multiple trajectories from a starting state to estimate win rates and optimize the policy.

Policy Optimization

By comparing these trajectories, the system estimates a "win rate" for specific actions. Using Proximal Policy Optimization (PPO), the neural network is updated to favor decisions that lead to "winning" trajectories. This allows the model to learn what a "better" outcome looks like through direct competition rather than being constrained by a fixed mathematical formula.

Experimental Evaluation

We validated Zwei across two distinct video transmission tasks:

  1. Adaptive Bitrate (ABR): Evaluated using HSDPA (mobile) and FCC (broadband) datasets on both HD and 4K video content.
  2. Crowd-sourced Live Streaming (CLS): Tested on a real-world CDN scheduling dataset spanning one week of operational data.

Results and Analysis

Zwei demonstrated significant performance improvements over state-of-the-art models such as Pensieve (DRL-based ABR) and RobustMPC.

ABR Performance

In 4K video scenarios, Zwei improved the Elo score by 32.24% compared to Pensieve. It successfully increased average bitrates by approximately 10% while simultaneously reducing rebuffering time by over 56%. As shown in the Pareto analysis below, Zwei consistently achieves a superior balance between bitrate and rebuffering across different network traces.

Performance comparison of Zwei against state-of-the-art methods (BOLA, Pensieve, RobustMPC) across Norway tram and metro network traces.

CLS Performance

In live streaming scheduling, Zwei reduced overall operational costs by 22% and decreased the stalling ratio by 6.5% compared to previous learning-based scheduling methods. The model also proved highly efficient, converging to state-of-the-art performance within approximately 200 training epochs.

Conclusion

Zwei represents a shift away from the fragile process of reward engineering toward a more robust, requirement-driven optimization. By leveraging self-play and simple priority rules, the framework faithfully optimizes policies for complex video transmission tasks. Future work will focus on incorporating additional metrics, such as bitrate switching smoothness, into the competition rules to further enhance the user experience.

Summary updated

Abstract

Video transmission services adopt adaptive algorithms to ensure users' demands. Existing techniques are often optimized and evaluated by a function that linearly combines several weighted metrics. Nevertheless, we observe that the given function fails to describe the requirement accurately. Thus, such proposed methods might eventually violate the original needs. To eliminate this concern, we propose \emph{Zwei}, a self-play reinforcement learning algorithm for video transmission tasks. Zwei aims to update the policy by straightforwardly utilizing the actual requirement. Technically, Zwei samples a number of trajectories from the same starting point and instantly estimates the win rate w.r.t the competition outcome. Here the competition result represents which trajectory is closer to the assigned requirement. Subsequently, Zwei optimizes the strategy by maximizing the win rate. To build Zwei, we develop simulation environments, design adequate neural network models, and invent training methods for dealing with different requirements on various video transmission scenarios. Trace-driven analysis over two representative tasks demonstrates that Zwei optimizes itself according to the assigned requirement faithfully, outperforming the state-of-the-art methods under all considered scenarios.
BibTeX
@inproceedings{huang2020self,
  author = {Tianchi Huang and Rui-Xiao Zhang and Lifeng Sun},
  title = {{Self-play Reinforcement Learning for Video Transmission}},
  booktitle = {NOSSDAV},
  year = {2020},
  url = {https://openreview.net/forum?id=ZgoCJoJLgmc},
}
Figure