Towards retina-quality VR video streaming: 15ms could save you 80% of your bandwidth

Luke Hsiao, Brooke Krajancich, Philip Alexander Levis, Gordon Wetzstein, Keith Winstein

TL;DR

Streaming retina-quality VR video is currently impossible due to massive bandwidth requirements. This paper demonstrates that by driving motion-to-photon latency below 15 ms, foveated video compression can shrink the required high-resolution gaze area, saving up to 80% of bandwidth compared to traditional encoding. This highlights ultra-low latency as a critical enabler for immersive, next-generation VR streaming.

The Challenge of Retina-Quality VR

To achieve truly immersive "retina-quality" VR, systems need to stream 360-degree video at massive resolutions (e.g., 5.7K or higher) and high frame rates. This requires gigabits of bandwidth, far exceeding current internet capabilities.

A popular workaround is foveated video compression, which exploits human visual acuity by rendering only the exact spot the user is looking at (the fovea) in high resolution, while heavily compressing the peripheral vision. However, there's a catch: motion-to-photon latency. If the system takes too long to update the display after the eye moves, the user's gaze will "escape" the high-resolution zone, and they will see the low-resolution periphery.

Foveated video compression must account for system latency. Higher latency requires a wider high-resolution region to prevent gaze escape, leading to 'wasted bits'.

To compensate for high latency, current systems must enlarge the high-resolution foveal region. As shown in Figure 1, this creates a buffer of "wasted bits" where the video quality is higher than the eye's acuity requires. Conversely, if the buffer is too small for the given latency, the user experiences "visual artifacts" as their gaze lands on low-quality regions.

Methodology: The Ultra-Low Latency Prototype

Because current commercial VR headsets (HMDs) have high latency (>45 ms) and sub-retina resolutions, the authors built a custom ultra-low-latency desktop prototype to serve as a proxy for future VR hardware. They used a 144 Hz 4K monitor, a high-speed Eyelink 1000 eye tracker, and a custom Rust-based client-server architecture communicating via shared memory to eliminate network overhead.

Instead of complex, computationally heavy algorithms that smoothly degrade video quality (like Gaussian blur maps), they used a simpler two-stream approach. The server encodes a low-resolution background frame and a small, high-resolution cropped frame centered on the user's gaze. The client decodes and blends them. This results in a "step" function foveation profile.

Comparison of foveation profiles. The authors utilize a 'step' function approach for its computational simplicity and efficiency at low latencies.

The authors conducted a user study where participants viewed videos at different artificial latencies (14 ms, 45 ms, 81 ms) to determine the minimum bitrate required to maintain perceived visual quality. They used two diverse 4K videos from Derf’s Test Media Collection: barscene (natural gaze guidance) and square_timelapse (frequent gaze jumps).

Key Insights: The Latency Cliff

The study revealed that the relationship between latency and required bitrate is non-linear. Reducing latency from 81 ms to 45 ms—the range of current high-end commercial HMDs—yielded negligible bandwidth savings.

The real breakthrough occurs at ultra-low latencies (sub-15 ms). At this point, the system hits a "latency cliff" where the required bitrate suddenly drops. Lower latency allows for a much smaller high-resolution buffer, leading to massive bandwidth savings. Furthermore, by driving latency down, the system can rely on the computationally simpler "two-stream" step-function approach rather than complex smooth degradation algorithms, saving valuable processing time.

Results and Performance

The custom prototype achieved an end-to-end motion-to-photon latency of ~14 ms. At this latency, the system achieved:

  • A 5x compression improvement (saving ~80% of bandwidth) compared to a standard 28 Mbit/s baseline, with no perceptible drop in quality.
  • A 2x reduction in bandwidth compared to a 45 ms latency system (representing current state-of-the-art HMDs) for the same perceived quality.

Limitations and Future Work

While promising, several hurdles remain for real-world deployment:

  • Network Latency: The prototype used shared memory. Real-world use requires edge servers within a few milliseconds of network round-trip time (RTT) to maintain the <15 ms budget.
  • Compute Cost: The encoder-in-the-loop approach requires real-time per-user encoding, which is more expensive than standard broadcasting.
  • Hardware Proxy: Findings need validation on actual VR HMDs once they reach the necessary resolution and latency specifications.

Future research will focus on testing over realistic edge-computing networks, integrating with other foveated graphics techniques, and expanding the user study to a wider variety of video content.

Summary updated

Abstract

Virtual reality systems today cannot yet stream immersive, retina-quality virtual reality video over a network. One of the greatest challenges to this goal is the sheer data rates required to transmit retina-quality video frames at high resolutions and frame rates. Recent work has leveraged the decay of visual acuity in human perception in novel gaze-contingent video compression techniques. In this paper, we show that reducing the motion-to-photon latency of a system itself is a key method for improving the compression ratio of gaze-contingent compression. Our key finding is that a client and streaming server system with sub-15ms latency can achieve 5x better compression than traditional techniques while also using simpler software algorithms than previous work.
BibTeX
@article{hsiao2022towards,
  author = {Luke Hsiao and Brooke Krajancich and Philip Alexander Levis and Gordon Wetzstein and Keith Winstein},
  title = {{Towards retina-quality VR video streaming: 15ms could save you 80\% of your bandwidth}},
  journal = {Comput. Commun. Rev.},
  year = {2022},
  doi = {10.1145/3523230.3523233},
}
Figure