Ten Ways in which Virtual Reality Differs from Video Streaming

Gustavo de Veciana, Sonia Fahmy, George Kesidis, Voicu PopescuarXiv 2024 · DOI · arXiv

Abstract

Traditional network architectures optimized for 2D video streaming are fundamentally ill-suited for the unique demands of Virtual Reality (VR). This paper identifies ten core differences—ranging from unpredictable user interaction to the reusability of 3D assets—providing a roadmap for rethinking rendering, caching, and transport protocols to meet the strict 10ms latency requirements of immersive environments.

Introduction: Beyond "Video with More Pixels"

For decades, the networking community has optimized for 2D video streaming platforms like Netflix and YouTube. These systems assume sequential viewing, predictable buffering, and stateless servers. However, VR is not merely "heavy video." It involves 6-Degrees-of-Freedom (6DoF) movement, real-time interaction with objects, and a heterogeneous mix of data types including meshes, point clouds, and spatial audio.

The central challenge is achieving a "motion-to-photon" latency of less than 10ms to prevent cybersickness. Current standard streaming stacks, designed for 2D content, struggle to bridge the gap between complex 3D data representations and real-time delivery requirements.

Data Representations and Level-of-Detail

Unlike 2D video, which consists of a sequence of pixel grids, VR environments are composed of geometric primitives. The authors utilize a massive Manhattan Urban Environment model, containing 3.7 million triangles, to illustrate the complexities of Level-of-Detail (LoD) adaptation.

Figure 1: A wireframe mesh representation of the Manhattan Urban Environment, illustrating the high geometric complexity of 3D assets.

In addition to meshes, point clouds are used for progressive refinement. As shown in the point cloud representations of the same environment, data can be "streamed" in layers of increasing density, allowing for immediate low-fidelity rendering that improves as more data arrives.

Figure 3: A point cloud representation of the urban environment, demonstrating an alternative data format used for progressive 3D streaming.

Key Architectural Shifts

The paper categorizes the differences between 2D streaming and VR into four major areas:

1. Data Reusability and Caching

In 2D video, frames are discarded once displayed. In VR, assets like building meshes are static and can be cached locally. As a user moves, previously downloaded assets remain relevant, transforming the streaming problem into one of distributed caching.

2. Out-of-Order Transport

Traditional video streaming relies on in-order delivery; losing packet #2 prevents the display of packet #3. In VR, if a packet containing a specific object (e.g., a tree) is lost, the rest of the world can still be rendered. The authors argue for out-of-order, partially reliable transport—likely using QUIC-based datagrams—to avoid head-of-line blocking.

3. Split Rendering and Predictive Padding

Headsets are often battery-constrained, while the cloud introduces too much lag. The "sweet spot" is split rendering: performing basic geometry processing on the headset while offloading heavy lighting or physics to a nearby edge server. Furthermore, because user movement (like "teleporting") is unpredictable, systems must use "predictive padding," sending extra data around the predicted field-of-view (FoV) to handle sudden head movements.

Conclusion and Future Directions

The authors conclude that VR requires a fundamental rethinking of the network stack. Designers must move away from TCP-like reliability toward application-level control over data obsolescence. Future research must focus on defining objective Quality of Experience (QoE) metrics that account for both visual fidelity and physical comfort, managing dynamic LoD across shared network cells, and establishing standardized benchmarks for 6DoF movement.

Summary updated

Abstract

Virtual Reality (VR) applications have a number of unique characteristics that set them apart from traditional video streaming. These characteristics have major implications on the design of VR rendering, adaptation, prefetching, caching, and transport mechanisms. This paper contrasts VR to video streaming, stored 2D video streaming in particular, and discusses how to rethink system and network support for VR.
BibTeX
@misc{veciana2024ten,
  author = {Gustavo de Veciana and Sonia Fahmy and George Kesidis and Voicu Popescu},
  title = {{Ten Ways in which Virtual Reality Differs from Video Streaming}},
  year = {2024},
  eprint = {2411.19730},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.19730},
}
Figure