Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality Streaming

Shaoyu Chen, Budmonde Duinkharjav, Xin Sun, Li-Yi Wei, Stefano Petrangeli, Jose Echevarria, Cláudio T. Silva, Qi Sun

Abstract: Media streaming has been adopted for a variety of applications such as entertainment, visualization, and design. Unlike video/audio streaming where the content is usually consumed sequentially, 3D applications such as gaming require streaming 3D assets to facilitate client-side interactions such as object manipulation and viewpoint movement. Compared to audio and video streaming, 3D streaming often requires larger data sizes and yet lower latency to ensure sufficient rendering quality, resolution, and latency for perceptual comfort. Thus, streaming 3D assets can be even more challenging than streaming audios/videos, and existing solutions often suffer from long loading time or limited quality. To address this critical and timely issue, we propose a perceptually-optimized progressive 3D streaming method for spatial quality and temporal consistency in immersive interactions. Based on the human visual mechanisms in the frequency domain, our model selects and schedules the streaming dataset for optimal spatial-temporal quality. We also train a neural network for our model to accelerate this decision process for real-time client-server applications. We evaluate our method via subjective studies and objective analysis under varying network conditions (from 3G to 5G) and client devices (HMD and traditional displays), and demonstrate better visual quality and temporal consistency than alternative solutions.

Introduction

Streaming high-quality 3D assets, such as complex meshes and textures, for Virtual Reality (VR) presents a significant technical challenge. Unlike 2D video, which is consumed linearly, VR requires interactive freedom. Users move their heads and manipulate objects, necessitating that the "edge" device (e.g., a Head-Mounted Display) has 3D data ready for local rendering.

Existing solutions typically fall into two categories: uniform streaming, which updates all assets simultaneously and leads to long loading times or pervasive blurriness; and basic foveated streaming, which prioritizes detail where the user is looking but often causes distracting "popping" artifacts (sudden jumps in quality) that the human eye is highly sensitive to. Instant Reality addresses these issues by using human perception models to prioritize which parts of a 3D scene to download first, delivering high-quality experiences with significantly reduced artifacts even on limited bandwidth.

Perceptually-Optimized Streaming Pipeline

The core of the system is a perceptually-optimized progressive streaming pipeline that decides exactly which 3D "bits" are most important at any given millisecond.

Spatio-Temporal Perception and Saccade-Awareness

The system utilizes a model based on Weber’s Law to calculate "perceived change," accounting for both where the user is looking (foveation) and how their eyes are moving. A key insight is the utilization of saccadic suppression. During a "saccade"—the rapid, tiny jumps the eyes make—human vision is effectively suppressed. The system detects these moments to perform aggressive updates that would otherwise be jarring, effectively using these intervals as a bandwidth "cheat code" to resolve popping artifacts before the eye regains focus.

Comparison of different streaming strategies: (a) edge view, (b) cloud data, (c) uniform streaming, (e) fixation-based, and (g) the proposed saccade-based streaming, with their respective quality heatmaps (d, f, h).

Neural Acceleration and Optimization

To map 2D retinal perception back to 3D primitives (triangles, voxels, or points), the authors use a deferred shading approach. Because calculating these complex perceptual models in real-time is computationally expensive, the system employs a lightweight Multi-Layer Perceptron (MLP). This neural accelerator predicts the importance of 3D assets based on camera and gaze vectors, reducing computation time from approximately 390ms to just 20ms. Finally, a greedy optimization algorithm treats streaming as a "knapsack problem," selecting the set of updates that provide the highest perceptual gain per bit within current bandwidth limits.

Evaluation and Results

The system was evaluated using various 3D data types, including triangle meshes (a "Warrior" character), displaced meshes (terrain), volume data (CT-scanned skeleton), and dynamic scenes.

Key results include:

  • Subjective Preference: In user studies, participants preferred the Instant Reality method over standard uniform streaming more than 90% of the time.
  • Performance: The neural accelerator enabled the system to run at a smooth 90 FPS on an NVIDIA RTX 2080, representing a 19x speedup in decision-making.
  • Efficiency and Robustness: Objective metrics (using FovVideoVDP) showed the system reaches high-quality thresholds significantly faster than alternatives across 3G, 4G, and 5G simulations. It also maintained high quality even with artificial latencies up to 175ms, proving robustness against real-world network jitter.

Limitations and Future Work

While effective, the current model requires scene-specific training for the MLP to maintain high accuracy. Additionally, because 3D units are updated independently, "cracks" can occasionally appear in meshes if levels of detail do not match perfectly. Future work will focus on integrating complex texture streaming and exploring applications in Augmented Reality (AR), where high-resolution requirements are even more demanding.

Summary updated

Abstract

Media streaming has been adopted for a variety of applications such as entertainment, visualization, and design. Unlike video/audio streaming where the content is usually consumed sequentially, 3D applications such as gaming require streaming 3D assets to facilitate client-side interactions such as object manipulation and viewpoint movement. Compared to audio and video streaming, 3D streaming often requires larger data sizes and yet lower latency to ensure sufficient rendering quality, resolution, and latency for perceptual comfort. Thus, streaming 3D assets can be even more challenging than streaming audios/videos, and existing solutions often suffer from long loading time or limited quality. To address this critical and timely issue, we propose a perceptually-optimized progressive 3D streaming method for spatial quality and temporal consistency in immersive interactions. Based on the human visual mechanisms in the frequency domain, our model selects and schedules the streaming dataset for optimal spatial-temporal quality. We also train a neural network for our model to accelerate this decision process for real-time client-server applications. We evaluate our method via subjective studies and objective analysis under varying network conditions (from 3G to 5G) and client devices (HMD and traditional displays), and demonstrate better visual quality and temporal consistency than alternative solutions.
BibTeX
@article{chen2022instant,
  author = {Shaoyu Chen and Budmonde Duinkharjav and Xin Sun and Li-Yi Wei and Stefano Petrangeli and Jose Echevarria and Cláudio T. Silva and Qi Sun},
  title = {{Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality Streaming}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2022},
  url = {https://openreview.net/forum?id=sQfRB_hHs_n},
}
Figure