Abstract
The popularity of immersive videos has prompted extensive research into neural adaptive tile-based streaming to optimize video transmission over networks with limited bandwidth. However, the diversity of users' viewing patterns and Quality of Experience (QoE) preferences has not been fully addressed yet by existing neural adaptive approaches for viewport prediction and bitrate selection. Their performance can significantly deteriorate when users' actual viewing patterns and QoE preferences differ considerably from those observed during the training phase, resulting in poor generalization. In this paper, we propose MANSY, a novel streaming system that embraces user diversity to improve generalization. Specifically, to accommodate users' diverse viewing patterns, we design a Transformer-based viewport prediction model with an efficient multi-viewport trajectory input output architecture based on implicit ensemble learning. Besides, we for the first time combine the advanced representation learning and deep reinforcement learning to train the bitrate selection model to maximize diverse QoE objectives, enabling the model to generalize across users with diverse preferences. Extensive experiments demonstrate that MANSY outperforms state-of-the-art approaches in viewport prediction accuracy and QoE improvement on both trained and unseen viewing patterns and QoE preferences, achieving better generalization.
Introduction
Streaming immersive (360-degree) video presents a significant bandwidth challenge, as these files are often 4–6 times larger than standard videos. To mitigate this, tile-based streaming is employed, where video is partitioned into a grid of "tiles." The system predicts the user's viewport and prioritizes high-definition transmission for those specific tiles. However, existing models often struggle with user diversity. Viewing patterns vary significantly between "explorers," who scan the entire scene, and "focusers," who remain centered on primary subjects. Furthermore, Quality of Experience (QoE) is subjective; some users prioritize resolution, while others prioritize smooth, rebuffering-free playback. Traditional Deep Reinforcement Learning (DRL) models often overfit to average patterns, leading to poor generalization in real-world scenarios.
Methodology
MTIO-Transformer for Viewport Prediction
To address diverse viewing patterns, MANSY utilizes a Multi-viewport Trajectory Input Output (MTIO) architecture. Instead of a standard single-output model, it employs a Transformer network with multiple "heads" to perform implicit ensemble learning. While traditional ensembles are computationally expensive for VR hardware, MANSY leverages the over-parameterization of modern neural networks to train multiple sub-networks within a single architecture. By averaging predictions from these heads, the model reduces individual biases and provides robust predictions across different user types.
RepL-based Bitrate Selection
To handle diverse QoE preferences, MANSY integrates DRL with Representation Learning (RepL). A secondary "QoE Identifier" network acts as a teacher, calculating the Mutual Information between the agent's bitrate decisions and the user's specific QoE preferences. This approach forces the DRL agent to learn the underlying relationship between its actions and the user's goals, preventing "catastrophic forgetting" where a model loses the ability to optimize for one preference while learning another. Residual connections are also utilized to ensure preference information persists through the deep network layers.
Evaluation and Results
The system was evaluated using the Wu2017 and Jin2022 datasets, which encompass a wide variety of video content and user trajectories.
Performance Metrics
MANSY demonstrated superior generalization compared to state-of-the-art (SOTA) baselines such as RLVA and DVMS:
- Viewport Prediction: Accuracy improved by 1.3%–5.2% on known patterns and 3.2%–8.8% on unseen patterns.
- Streaming Quality: Average QoE scores increased by 3.0%–14.1% for trained preferences and 3.2%–15.3% for unseen preferences.
Computational Efficiency
Despite its ensemble-based approach, MANSY maintains a low overhead. Testing on an NVIDIA 3090 showed a negligible increase in memory consumption (0.25%) and inference time (1.56%) compared to single-head models, making it viable for real-time applications.
Discussion and Future Work
MANSY successfully addresses the generalization gap in immersive video streaming by embracing user diversity. Key insights include the effectiveness of using mutual information as a reward signal and the utility of residual learning for preference stability.
Future research will explore adaptive weighting for the Transformer heads to further refine accuracy for specific users. Additionally, the team aims to develop methods for automatically inferring user preferences from interaction data and testing the system on mobile VR chipsets, such as the Snapdragon XR2, to ensure performance on constrained hardware.
Summary updated