Improving ABR Performance for Short Video Streaming Using Multi-Agent Reinforcement Learning with Expert Guidance

Yueheng Li, Qianyuan Zheng, Zicheng Zhang, Hao Chen, Zhan Ma

Abstract

Short video streaming platforms like TikTok and Reels present unique challenges for Adaptive Bitrate (ABR) algorithms due to rapid user swiping behavior and the need for proactive prefetching. Traditional ABR methods, optimized for long-form content, often fail to balance buffer management with bitrate selection in these dynamic environments. This paper introduces Incendio, a framework that utilizes Multi-Agent Reinforcement Learning (MARL) combined with imitation learning from expert rules. By decomposing the decision-making process into specialized agents and bootstrapping training with expert guidance, Incendio achieves a 53.2% improvement in utility scores over state-of-the-art methods while maintaining the low computational overhead required for mobile deployment.

Introduction

In the landscape of modern mobile internet, short video streaming has become a dominant medium. Unlike traditional long-form video services (e.g., Netflix or YouTube), short video applications are characterized by high-frequency user interactions, specifically the "swipe" gesture. This behavior necessitates a complex decision-making process: the system must not only decide the quality (bitrate) of the current video but also determine which future videos to prefetch and how much data to buffer.

Existing Reinforcement Learning (RL) approaches to ABR often struggle with the massive action space created by these dual requirements, leading to slow convergence and sub-optimal performance. To address this, we present Incendio, a hierarchical multi-agent framework designed to optimize the Quality of Experience (QoE) for short video consumers.

The Short Video Streaming Challenge

Traditional ABR algorithms focus primarily on Bitrate Adaptation (BA). However, in short video contexts, Buffer Management (BM) is equally critical. If the system prefetches too aggressively, it wastes significant bandwidth when a user swipes away early. Conversely, insufficient prefetching leads to rebuffering events that degrade user experience.

This creates a "curse of dimensionality" for single-agent RL models. Attempting to learn both which video to download and at what quality simultaneously results in an inefficient exploration of the action space. Furthermore, the high computational cost of traditional Model Predictive Control (MPC) methods makes them difficult to implement on mobile devices with limited processing power.

The Incendio Framework

Incendio addresses these challenges through three primary innovations: hierarchical decomposition, a two-stage training pipeline, and a specialized neural architecture.

Hierarchical Multi-Agent Design

Instead of a monolithic controller, Incendio splits the task between two specialized agents:

  1. BM-agent (Buffer Management): This agent manages the download queue. It decides which video ID to fetch next or instructs the system to enter a "sleep" state to conserve resources and prevent bandwidth waste.
  2. BA-agent (Bitrate Adaptation): Once the BM-agent selects a video, the BA-agent determines the optimal bitrate level based on current network conditions and buffer status.

Two-Stage Training with Expert Guidance

To overcome the slow start typical of RL, Incendio employs a two-stage training strategy:

  • Stage 1: Imitation Learning (IL): The agents are initially trained to mimic PDAS, a high-performing but computationally intensive rule-based algorithm. This "expert guidance" provides a warm start, ensuring the agents begin with a baseline of competent logic.
  • Stage 2: MARL Fine-tuning: The agents are then refined using Multi-Agent Proximal Policy Optimization (MAPPO). During this phase, they learn to collaborate to maximize a shared utility function that balances video quality against rebuffering and bandwidth wastage.

Neural Architecture

The framework utilizes Gated Recurrent Units (GRU) to model temporal network fluctuations and 2D Convolutional Neural Networks (2D-CNNs) to process spatial relationships within the video queue, such as user retention trends and video metadata.

Experimental Evaluation

The researchers evaluated Incendio using the MMGC2022 and DUASVS datasets, which provide real-world user retention rates and video statistics. Network simulations were conducted using the Oboe and FCC datasets to replicate diverse mobile environments (4G, 5G, and Wi-Fi).

Key Results

  • Utility Gains: Incendio outperformed the state-of-the-art PDAS algorithm by 53.2% in overall utility.
  • Efficiency: The framework reduced rebuffering by 39.1% – 61.8% and slashed bandwidth wastage by 34.9% – 59.2% compared to existing baselines.
  • Deployment Readiness: Unlike MPC-based methods that scale poorly with queue size, Incendio maintains a near-constant inference time of approximately 1ms, making it ideal for real-time execution on mobile hardware.

Discussion and Future Work

A critical insight from this work is the importance of Retention-Aware Rewards. By incorporating the likelihood of a user swiping away into the reward function, Incendio learns to deprioritize downloads for videos with low predicted retention, directly addressing the primary cause of bandwidth waste in short video apps.

While highly effective, the current iteration of Incendio uses a fixed sleep duration (200ms) and relies on externally provided retention models. Future research will explore dynamic sleep intervals and integrated user behavior prediction to further refine the system's efficiency.

Conclusion

Incendio demonstrates that by combining the strengths of expert-driven imitation learning with the flexibility of multi-agent reinforcement learning, it is possible to significantly improve the performance of short video streaming. The framework offers a scalable, efficient, and high-performance solution for the next generation of mobile content delivery.

Summary updated

BibTeX
@inproceedings{li2023improving,
  author = {Yueheng Li and Qianyuan Zheng and Zicheng Zhang and Hao Chen and Zhan Ma},
  title = {{Improving ABR Performance for Short Video Streaming Using Multi-Agent Reinforcement Learning with Expert Guidance}},
  booktitle = {NOSSDAV},
  year = {2023},
  url = {https://openreview.net/forum?id=ZkHuykRXoc},
}
Figure