Content-Aware Personalised Rate Adaptation for Adaptive Streaming via Deep Video Analysis

Guanyu Gao, Linsen Dong, Huaizheng Zhang, Yonggang Wen, Wenjun Zeng

Abstract

Adaptive bitrate (ABR) streaming is the standard for delivering smooth video under fluctuating network conditions. However, most current ABR algorithms are content-agnostic, ignoring the semantic importance of video segments. This paper introduces a Content-of-Interest (CoI) based rate adaptation scheme. By utilizing deep learning to identify "interesting" video segments and a Deep Q-Network (DQN) to prioritize their quality, the system improves subjective viewer experience without sacrificing traditional network performance metrics.

Introduction

Adaptive Bitrate (ABR) streaming, used by platforms like YouTube and Netflix, switches quality levels to prevent buffering. A significant limitation of existing ABR logic is that it treats all content equally; a high-action goal in a soccer match is given the same priority as a static shot of the crowd. From a user's perspective, pixelation during a critical moment is far more disruptive than during a mundane one. The authors argue that "interestingness"—the level of subjective engagement a segment draws—should dictate the allocation of the limited "bitrate budget."

System Framework

The proposed framework operates in two main stages: offline semantic analysis and online rate adaptation. The interaction between the server-side content analysis and the client-side player is central to this approach.

Figure 1: Overview of the CoI-based rate adaptation system architecture, showing the interaction between the Streaming Server and the Video Player.
  1. Interestingness Recognition: Using 3D Convolutional Networks (3D ConvNets), the system analyzes spatiotemporal features (movement over time) rather than just static frames. The model extracts features from video chunks and outputs a real-valued "interestingness score."
  2. DQN-Based Rate Adaptation: Rate adaptation is formulated as a Reinforcement Learning (RL) problem where a Deep Q-Network (DQN) agent selects the optimal bitrate for the next chunk.

DQN-Based Rate Adaptation

The DQN agent is trained to make decisions based on a multi-dimensional state that incorporates both network conditions and content semantics.

Figure 3: The DQN architecture, illustrating the state inputs (including interestingness) and the resulting bitrate selection (actions).

As shown in the architecture, the agent considers predicted bandwidth, current buffer occupancy, the previous bitrate, and the interestingness scores of upcoming chunks. The "secret sauce" is a modified reward function that weights video quality by its interestingness score. If the agent delivers high quality during an "interesting" segment, it receives a significantly higher reward, teaching it to "save" bandwidth for high-stakes moments.

Key Results and Insights

The system was evaluated using the TVSum dataset for interestingness and realistic bandwidth traces (FCC and 3G/HSDPA) for network simulation. Key findings include:

  • High Precision: The interestingness recognition model achieved a Mean Squared Error (MSE) of 0.02, accurately reflecting human engagement levels.
  • Semantic Alignment: Unlike standard methods (BBA or Robust-MPC), the CoI method showed a strong positive correlation between content interestingness and allocated bitrate.
  • No Technical Penalty: The system achieved content-awareness without a significant penalty to objective metrics like rebuffering time or bitrate variation. In fact, the CoI method reached the highest mean average bitrate per session among all tested baselines.

Conclusion and Future Work

By treating interestingness as a multiplier for quality in the RL reward function, the agent naturally learns to prioritize critical content. While the current model uses an average human interest score, future work could involve personalizing adaptation to individual preferences and optimizing the DQN for low-power mobile devices.

Summary updated

Abstract

Adaptive bitrate (ABR) streaming is the de facto solution for achieving smooth viewing experiences under unstable network conditions. However, most of the existing rate adaptation approaches for ABR are content-agnostic, without considering the semantic information of the video content. Nevertheless, semantic information largely determines the informativeness and interestingness of the video content, and consequently affects the QoE for video streaming. One common case is that the user may expect higher quality for the parts of video content that are more interesting or informative so as to reduce video distortion and information loss, given that the overall bitrate budgets are limited. This creates two main challenges for such a problem: First, how to determine which parts of the video content are more interesting? Second, how to allocate bitrate budgets for different parts of the video content with different significances? To address these challenges, we propose a Content-of-Interest (CoI) based rate adaptation scheme for ABR. We first design a deep learning approach for recognizing the interestingness of the video content, and then design a Deep Q-Network (DQN) approach for rate adaptation by incorporating video interestingness information. The experimental results show that our method can recognize video interestingness precisely, and the bitrate allocation for ABR can be aligned with the interestingness of video content while not compromising the performances on objective QoE metrics.
BibTeX
@inproceedings{gao2019content,
  author = {Guanyu Gao and Linsen Dong and Huaizheng Zhang and Yonggang Wen and Wenjun Zeng},
  title = {{Content-Aware Personalised Rate Adaptation for Adaptive Streaming via Deep Video Analysis}},
  booktitle = {ICC},
  year = {2019},
  doi = {10.1109/icc.2019.8761156},
}
Figure