Abstract
Video transmission services adopt adaptive algorithms to ensure users' demands. Existing techniques are often optimized and evaluated by a function that linearly combines several weighted metrics. Nevertheless, we observe that the given function fails to describe the requirement accurately. Thus, such proposed methods might eventually violate the original needs. To eliminate this concern, we propose Zwei, a self-play reinforcement learning algorithm for video transmission tasks. Zwei aims to update the policy by straightforwardly utilizing the actual requirement. Technically, Zwei samples a number of trajectories from the same starting point and instantly estimates the win rate w.r.t the competition outcome. Here the competition result represents which trajectory is closer to the assigned requirement. Subsequently, Zwei optimizes the strategy by maximizing the win rate. To build Zwei, we develop simulation environments, design adequate neural network models, and invent training methods for dealing with different requirements on various video transmission scenarios. Trace-driven analysis over two representative tasks demonstrates that Zwei optimizes itself according to the assigned requirement faithfully, outperforming the state-of-the-art methods under all considered scenarios.
Introduction
In modern video streaming, maintaining a high Quality of Experience (QoE) is a complex balancing act. Traditional Adaptive Bitrate (ABR) algorithms rely on Deep Reinforcement Learning (DRL) trained via hand-tuned reward functions. These functions typically combine metrics like bitrate and rebuffering time into a single scalar value using linear weights. However, this approach often fails because fixed weights cannot capture the deterministic priorities of human users across diverse network conditions (e.g., a stable fiber connection versus a fluctuating mobile signal). When the reward function is misaligned with actual user requirements, the AI learns sub-optimal strategies that may technically maximize a score while providing a poor viewing experience.
Methodology: The Zwei Framework
To overcome the limitations of reward engineering, we introduce Zwei, a framework that utilizes self-play to optimize policies directly against user requirements. Instead of chasing a numerical reward, Zwei learns through comparison and competition.
Self-Play and Trajectory Sampling
Zwei employs a Monte Carlo sampling approach to generate multiple potential "trajectories" (sequences of bitrate decisions) from a single starting point in a network trace. These trajectories are then compared in pairs using a deterministic "Rule" that reflects actual user priorities (e.g., "minimize rebuffering first, then maximize bitrate").
Policy Optimization
By comparing these trajectories, the system estimates a "win rate" for specific actions. Using Proximal Policy Optimization (PPO), the neural network is updated to favor decisions that lead to "winning" trajectories. This allows the model to learn what a "better" outcome looks like through direct competition rather than being constrained by a fixed mathematical formula.
Experimental Evaluation
We validated Zwei across two distinct video transmission tasks:
- Adaptive Bitrate (ABR): Evaluated using HSDPA (mobile) and FCC (broadband) datasets on both HD and 4K video content.
- Crowd-sourced Live Streaming (CLS): Tested on a real-world CDN scheduling dataset spanning one week of operational data.
Results and Analysis
Zwei demonstrated significant performance improvements over state-of-the-art models such as Pensieve (DRL-based ABR) and RobustMPC.
ABR Performance
In 4K video scenarios, Zwei improved the Elo score by 32.24% compared to Pensieve. It successfully increased average bitrates by approximately 10% while simultaneously reducing rebuffering time by over 56%. As shown in the Pareto analysis below, Zwei consistently achieves a superior balance between bitrate and rebuffering across different network traces.
CLS Performance
In live streaming scheduling, Zwei reduced overall operational costs by 22% and decreased the stalling ratio by 6.5% compared to previous learning-based scheduling methods. The model also proved highly efficient, converging to state-of-the-art performance within approximately 200 training epochs.
Conclusion
Zwei represents a shift away from the fragile process of reward engineering toward a more robust, requirement-driven optimization. By leveraging self-play and simple priority rules, the framework faithfully optimizes policies for complex video transmission tasks. Future work will focus on incorporating additional metrics, such as bitrate switching smoothness, into the competition rules to further enhance the user experience.
Summary updated