Abstract
In the digital era, Extended Reality (XR) is considered the next frontier. However, XR systems are computationally intensive and must operate within strict latency constraints. XR devices with finite computing resources are often limited in the Quality of Experience (QoE) they can offer, particularly when handling large-scale 3D data. This research addresses these limitations by offloading intensive rendering tasks to a remote server. We propose a remote rendering-enabled XR system that visualizes a high-fidelity 3D model of New York City on the Microsoft HoloLens 2. Our experimental results indicate that remote rendering significantly outperforms local rendering, improving average QoE by at least 21%. Furthermore, we characterize the network traffic patterns of the system developed under the OpenXR standard.
Introduction
Extended Reality (XR) devices, such as the Microsoft HoloLens 2, are designed as "mobile-first" systems. While this portability allows for seamless integration into various environments, it necessitates a reliance on finite onboard computing power and battery life. These constraints become apparent when attempting to visualize "Big 3D Data"—complex, high-polygon models like entire cityscapes or intricate industrial machinery.
When complex scenes are rendered locally on a headset, the frame rate often drops, leading to jitter, while increased latency can cause motion sickness. To overcome these hardware limitations, this study explores the efficacy of Remote Rendering. By treating the headset as a thin client that transmits tracking data to a high-performance server, the system can leverage powerful GPUs to render frames and stream them back as a video feed. The primary challenge is ensuring that the network latency introduced by this round-trip does not negate the graphical performance gains.
System Architecture and Methodology
The research team developed a comprehensive end-to-end remote rendering pipeline consisting of several key components:
1. Hardware and Software Integration
The system utilizes a Windows 10 server equipped with an Intel i9-11900F CPU and an NVIDIA RTX 3090 GPU to handle logic and high-intensity rendering. The HoloLens 2 serves as the display and input device, utilizing the OpenXR standard to manage the incoming video stream and user interactions.
2. Multi-Modal Interaction
To ensure a natural user experience within the 3D NYC model, three interaction models were implemented:
- Hand Tracking: Supports "near" interaction (direct touch) and "far" interaction (ray-casting and point-and-commit).
- Eye Gaze: Provides contextual feedback based on the user's focal point.
- Voice Commands: Enables hands-free navigation, such as jumping to specific landmarks like "Times Square."
3. QoE and Traffic Modeling
The study proposed a time-windowed Quality of Experience (QoE) metric. This mathematical model weighs frame rate smoothness and resolution quality against a "latency penalty" that increases exponentially as delays grow. Additionally, the researchers analyzed UDP network traffic using Wireshark and Scapy, employing an ARMA (Autoregressive Moving Average) model to predict future video frame sizes, which is essential for proactive bandwidth management.
Datasets
The primary visualization dataset was the ArcGIS Living Atlas of the World, specifically the 3D city model of New York City. This model includes layers for topography, imagery, streets, and informational layers such as population density and subway routes. To validate the traffic models, the authors generated a secondary dataset of network traces (packet sizes and intervals) captured during active XR sessions.
Key Insights
- Stability Over Raw Speed: While local rendering may occasionally reach acceptable speeds, it is inherently unstable. Remote rendering provides a consistent 55–60 FPS, which is critical for preventing VR-induced nausea.
- The Latency Trade-off: Although remote rendering introduces network delay, the total latency is often lower than local rendering for complex scenes. This is because the server completes the rendering task significantly faster than the mobile hardware.
- Predictable Traffic Patterns: Analysis revealed that XR network traffic follows a Normal distribution more closely than Laplace or Logistic distributions. This predictability allowed the ARMA(5, 4) model to accurately capture the autocorrelation structure of the video stream.
Experimental Results
The performance comparison between local and remote rendering yielded several significant findings:
- Frame Rate: At a resolution of 2048x1080, local rendering performance dropped to 9–21 FPS, whereas remote rendering maintained a stable 55–60 FPS.
- QoE Scores: The remote rendering system improved the average QoE score from 0.46 to 0.92 (on a 0–1 scale) for high-resolution tasks. Even in lower-resolution scenarios, remote rendering outperformed local rendering by 21%.
- Latency Management: Local high-quality rendering experienced latency spikes up to 110ms. In contrast, the remote rendering system remained within a more manageable range of 68–86ms.
Limitations and Future Work
Despite the improvements, the system currently sits above the 20ms "Motion-to-Photon" (MTP) threshold required for truly imperceptible latency. Furthermore, high-quality XR is extremely data-intensive; the authors estimate that raw 24-bit video at 60Hz/2K resolution would require 2.96 Gbps. Future research will focus on more aggressive compression techniques and optimizations for 5G/6G networks to handle such high throughput in real-world environments with variable network jitter.
Summary updated