Adaptive Streaming of 360-Degree Videos With Reinforcement Learning

Sohee Park, Minh Hoai, Arani Bhattacharya, Samir R. DasWACV 2021

TL;DR

This paper introduces ATRIA, a novel reinforcement learning (RL) based adaptive streaming system for 360-degree videos. By dynamically learning optimal tile selection and quality, ATRIA significantly improves user-perceived quality of experience (QoE) and bitrate compared to existing methods, even under fluctuating network conditions.

What Problem Does This Paper Address?

Streaming 360-degree videos presents a unique challenge: users only view a small portion (viewport) of the massive amount of data required to render the entire spherical scene. This leads to significant bandwidth waste if not managed efficiently. Existing adaptive streaming techniques for 360-degree content often rely on simplified rules or fixed strategies for selecting video tiles and their quality levels. These methods struggle to adapt effectively to the dynamic nature of user viewports and unpredictable network bandwidth fluctuations, resulting in suboptimal user experience (QoE), characterized by low perceived video quality and frequent stalls. The core problem is developing a streaming strategy that can intelligently balance bandwidth constraints, viewport prediction uncertainty, and the need for high-quality, smooth playback in real-time.

Methods

The paper proposes ATRIA (Adaptive streaming using ReInforcement leArning), an RL-based approach that treats adaptive streaming as a sequential decision-making problem.

  • Reinforcement Learning Framework: ATRIA utilizes a deep reinforcement learning agent (specifically, an actor-critic architecture inspired by A3C) to learn a streaming policy. The agent's goal is to maximize the cumulative reward, which is designed to reflect the user's Quality of Experience (QoE).
  • State Representation: The agent's state includes crucial information such as past network throughputs and download times, tile sizes across different quality levels, predicted tile view probabilities, current tile quality selections, previously downloaded segment qualities, remaining segments, and the player's buffer size.
  • Action Space: The paper introduces two action space variations:
  • ATRIA (Adaptive Tile Ordering): The agent decides both which tile to download and its quality level in a single action. This offers greater flexibility but results in a larger action space.
  • ATRIA-2 (Fixed Tile Ordering): The agent only decides the quality level for a pre-determined tile order. This simplifies the action space and leads to faster convergence.
  • Reward Function: The reward is a composite measure of user QoE, incorporating:
  • Viewer Perceived Video Bitrate: Only counts bitrates of tiles that overlap with the predicted viewport.
  • Rebuffering: Penalizes time spent stalled due to insufficient buffer.
  • Quality Smoothness: Penalizes fluctuations in perceived bitrate between consecutive segments and within the viewport.
  • Penalties: Negative rewards are given for re-downloading a tile at a lower quality or for missing tiles that overlap with the viewport.
  • Viewport Prediction: ATRIA leverages an external 3DCNN-based viewport prediction module, which uses saliency maps, motion maps, and head tracking history to estimate the probability of each tile being viewed.

Datasets

  • Network Traces: The system is trained and evaluated using approximately 300 real network traces from public datasets, including the broadband dataset and a mobile dataset. These traces are used to emulate diverse network conditions.
  • Video Content: Ten one-minute 360-degree videos are used for training and testing. Each video is encoded at six different bitrates (0.512 to 20 Mbps) and split into two-second segments.
  • Head Tracking Data: Head tracking data from 50 viewers for each video is used as ground truth for evaluating viewing experience, not as direct input to the streaming algorithm.

Key Insights and Ideas

  • RL for Dynamic Adaptation: The core insight is that the complex, sequential nature of adaptive 360-degree video streaming is well-suited for reinforcement learning, allowing for dynamic adaptation beyond static rules.
  • Managing Large Action Spaces: The paper addresses the challenge of a large action space in 360-degree streaming by proposing two strategies: a flexible but complex ATRIA and a simplified ATRIA-2, demonstrating that even a constrained action space can yield significant improvements.
  • Delayed Rewards and QoE: The reward function is designed to be evaluated per segment, reflecting the delayed nature of QoE assessment (which requires a full segment to be downloaded and played). This is crucial for effective RL training.
  • Generalization: ATRIA demonstrates strong generalization capabilities, performing well on unseen videos and even when trained on a single video, suggesting its learned policy is robust to content variations.

Key Results

  • Quantitative Performance:
    • ATRIA and ATRIA-2 achieve significantly higher average QoE than baseline methods (Mosaic, Flare, DRL360) by 1.6 to 3.4 times.
  • The average user-perceived bitrate is 1.3 to 1.7 times higher for ATRIA/ATRIA-2 compared to baselines.
  • Rebuffering ratios are comparable across methods, but ATRIA/ATRIA-2 excel in perceived bitrate and overall QoE.
  • Qualitative Findings (User Study):
    • Subjective user studies confirm the superiority of ATRIA and ATRIA-2, with ATRIA-2 consistently receiving the highest ratings across categories like overall quality, image color, and compression quality.
  • Visual comparisons show that DRL360 exhibits noticeable poor image quality in specific viewport areas, which is mitigated by ATRIA-2.

Limitations and Future Work

  • Smoothness within Viewport: While ATRIA and ATRIA-2 achieve high bitrates and overall QoE, their smoothness within the viewport is not always the best compared to some baselines. The authors suggest improving this aspect as a future direction.
  • Action Space Complexity: While ATRIA-2 simplifies the action space, the full ATRIA approach still faces challenges with very large action spaces, which could be further explored.
  • Viewport Prediction Accuracy: The performance of ATRIA is inherently tied to the accuracy of the viewport prediction module. Further improvements in prediction accuracy could lead to even better streaming performance.

Summary updated

BibTeX
@inproceedings{park2021adaptive,
  author = {Sohee Park and Minh Hoai and Arani Bhattacharya and Samir R. Das},
  title = {{Adaptive Streaming of 360-Degree Videos With Reinforcement Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure