GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System

Zhiye Tang, Qiudan Zhang, Lei Zhang, Junhui Hou, You Yang, Xu Wang

TL;DR

GSStream is the first end-to-end streaming system designed specifically for 3D Gaussian Splatting (3DGS). By combining a collaborative viewport prediction module with a deep reinforcement learning-based bitrate adapter, it achieves high-fidelity volumetric rendering over constrained networks, significantly outperforming existing point-cloud-based streaming methods.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has taken the computer graphics world by storm, offering photorealistic, real-time rendering of complex 3D scenes. However, this visual fidelity comes at a high cost: data volume. A single high-quality indoor scene can require hundreds of megabytes of storage.

For applications like VR/AR and remote digital twins, we need to stream this data in real-time. Standard streaming approaches fail here because:

  1. Bandwidth Intensity: You cannot send the entire scene at once.
  2. Viewport Sensitivity: To save bandwidth, you only want to send high-quality data for what the user is actually looking at (the Field of View), but predicting human head movement in 3D space is notoriously difficult.
  3. Scene Variability: Unlike 2D video, 3D scenes vary wildly in geometry and "tile" counts, making it hard to design a one-size-fits-all bitrate adaptation algorithm.

Methods

The authors introduce GSStream, which breaks the streaming pipeline into four main stages:

  1. Pre-processing: The scene (based on the Scaffold-GS format) is partitioned into 3D cubic tiles. Each tile is downsampled into six different quality levels using voxel grid filtering.
  2. Collaborative Viewport Prediction (CVP): Instead of just looking at the current user's history, this module uses Collaborative Priors. It learns from a dataset of how other users explored the same or similar scenes. It utilizes an iTransformer to model temporal dependencies and cross-attention mechanisms to fuse individual and group behaviors.
  3. DRL-based Bitrate Adaptation (DBA): This module decides which tiles to send at which quality level. It uses a Deep Deterministic Policy Gradient (DDPG) algorithm.
  4. Handling Space Variability: To solve the problem of different scenes having different numbers of tiles, the authors borrowed Set Abstraction (SA) and Feature Propagation (FP) modules from PointNet++. This allows the neural network to process tiles as an "unordered set" rather than a fixed-size vector, making the system scene-agnostic.

Datasets

The researchers built and released the first comprehensive user viewport trajectory dataset specifically for 3DGS content.

  • Content: 15 diverse volumetric scenes (8 outdoor, 7 indoor) sourced from Mip-NeRF 360, Deep Blending, and Tanks&Temples.
  • Participants: 32 subjects who freely explored the scenes using an HTC VIVE Pro Eye HMD.
  • Scale: Over 864,000 frames of trajectory data.
  • Finding: They noted that 3DGS users are much more "active" than users of traditional point-cloud video, showing 9x higher positional variance.

Key Insights and Ideas

  • The "Collaborative" Advantage: The authors realized that human curiosity is somewhat predictable. If ten people stopped to look at a specific statue in a virtual museum, the eleventh person likely will too. Fusing these "collaborative priors" with the user's own "historical priors" drastically reduces prediction error.
  • Tiles as Sets, Not Grids: Most streaming systems assume a fixed grid of tiles. By treating tiles as an unordered set and using PointNet-style architectures, GSStream can handle any scene geometry without retraining the underlying DRL model.
  • Importance-Aware Streaming: Not all Gaussians are equal. The system calculates an "importance score" for tiles based on the scaling factors of the anchors within them, ensuring that the most visually significant parts of the scene are prioritized during transmission.

Key Results

GSStream was tested against three major baselines: ViVo (rule-based), CaV3 (transformer-based), and GS3D (distance-based).

  • Visual Quality: GSStream achieved a mean Viewport SSIM (Structural Similarity Index) that was 118.9% higher than ViVo and roughly 10% higher than CaV3 and GS3D across various bandwidth constraints (40Mbps to 120Mbps).
  • Efficiency: Under a 120Mbps constraint on the "stump" scene, GSStream maintained an SSIM of 0.91, while the next best competitor (CaV3) fluctuated and dropped as low as 0.54 during rapid viewport transitions.
  • Bandwidth Stability: The system demonstrated much smoother and more efficient utilization of available bandwidth compared to the "bursty" nature of baseline methods.

Limitations and Future Work

  • Static Scenes Only: Currently, GSStream is optimized for static 3D scenes. The authors acknowledge that moving to dynamic volumetric video (4D) is the next major hurdle.
  • Codec Integration: The system currently uses voxel downsampling for quality levels. The authors suggest that developing specialized 3DGS-native codecs (similar to Draco for point clouds) could provide even greater bandwidth savings.
  • Computational Overhead: While the rendering is real-time, the DRL and Transformer-based prediction modules add computational load to the client/server that needs further optimization for mobile devices.

Summary updated

Abstract

Recently, the 3D Gaussian splatting (3DGS) technique for real-time radiance field rendering has revolutionized the field of volumetric scene representation, providing users with an immersive experience. But in return, it also poses a large amount of data volume, which is extremely bandwidth-intensive. Cutting-edge researchers have tried to introduce different approaches and construct multiple variants for 3DGS to obtain a more compact scene representation, but it is still challenging for real-time distribution. In this paper, we propose GSStream, a novel volumetric scene streaming system to support 3DGS data format. Specifically, GSStream integrates a collaborative viewport prediction module to better predict users' future behaviors by learning collaborative priors and historical priors from multiple users and users' viewport sequences and a deep reinforcement learning (DRL)-based bitrate adaptation module to tackle the state and action space variability challenge of the bitrate adaptation problem, achieving efficient volumetric scene delivery. Besides, we first build a user viewport trajectory dataset for volumetric scenes to support the training and streaming simulation. Extensive experiments prove that our proposed GSStream system outperforms existing representative volumetric scene streaming systems in visual quality and network usage. Demo video: https://youtu.be/3WEe8PN8yvA.
BibTeX
@misc{tang2026gsstream,
  author = {Zhiye Tang and Qiudan Zhang and Lei Zhang and Junhui Hou and You Yang and Xu Wang},
  title = {{GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System}},
  year = {2026},
  eprint = {2603.09718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.09718},
}
Figure