BOXR: Body and head motion Optimization framework for eXtended Reality

Cong Liu, Ziliang Zhang, Zexin Li, Hyoseung Kim, Cong Liu 0005

TL;DR

This paper introduces BOXR, a novel framework that optimizes both head and body motion latency in Extended Reality (XR) systems. By introducing a new metric (C2D latency) and a contention-preventive scheduling policy, BOXR significantly reduces motion sickness and improves the immersive experience, outperforming state-of-the-art methods.

What Problem Does This Paper Address?

Extended Reality (XR) systems, which include virtual, augmented, and mixed reality, aim to create immersive experiences by accurately tracking user movements and updating the displayed scene in real-time. A critical factor for a comfortable XR experience is minimizing the delay between a user's motion and its visual representation on the display. This delay, known as Motion-to-Display (M2D) latency, is particularly problematic for head movements, which are frequent and subtle. However, existing XR systems primarily focus on optimizing M2D latency, often neglecting the impact of larger, less frequent body movements. This oversight can lead to motion sickness, especially when users engage in activities involving significant body motion, such as walking or sports. The paper highlights that current M2D metrics only account for head motion captured by IMUs, failing to capture the latency introduced by body motion that requires camera-based localization.

Methods

BOXR introduces a comprehensive framework to address the limitations of existing XR systems by co-optimizing both head and body motion latencies. Key components include:

  • Camera-to-Display (C2D) Latency Metric: The paper defines a new metric, C2D latency, which specifically measures the delay introduced by body motion, from camera capture to display update. This complements the existing M2D (head motion) metric.
  • Contention-Preventive Scheduling Policy: BOXR implements a novel scheduling policy designed to prevent conflicts between critical tasks like scene reconstruction (SR) and asynchronous timewarp reprojection (ATWR). This policy ensures that the raw pose data used for display updates remains fresh, thereby reducing latency and improving consistency.
  • On-Demand IMUi: To optimize IMU data processing, the IMU integration (IMUi) task is triggered only when needed for SR and ATW jobs, rather than processing every IMU reading. This reduces wasted computation while still providing timely fused pose information.
  • Motion-Driven Visual Inertial Odometer (MVIO): To handle dynamic user motion (C1), MVIO dynamically adjusts the input image processing for the Visual Inertial Odometry (VIO) task. It achieves this by cropping the input camera image and adjusting the pyramid level of the feature extraction network based on a calculated "motion score." This helps to keep VIO execution times within a budget without significantly degrading pose quality.
  • Scene-Dependent Foveated Rendering (SFR): To address scene changes (C2), BOXR employs SFR, which dynamically adjusts the foveation area based on the number of objects in the viewport. By centering the foveation on object centroids, it prioritizes rendering resources where they are most needed, reducing rendering time without a significant loss in visual quality.

Datasets

The paper utilizes the following datasets for evaluation:

  • EuRoC MAV Datasets: This dataset is used for controlled trajectory experiments, providing visual-inertial data from indoor environments with varying motion levels.
  • KITTI Dataset: Used for evaluating the MVIO component in outdoor, large-motion scenarios.
  • Real-world Testcases: The authors created three XR applications (Video Watch, Room Explore, Escape Game) using the Godot game engine on a PC with a Zed Mini camera and Northstar Next HMD to evaluate BOXR in practical, real-world scenarios with varying magnitudes of user motion.

Key Insights and Ideas

  • The "Motion" Dichotomy: The paper effectively distinguishes between "head motion" (captured by IMU) and "body motion" (requiring camera localization), highlighting that optimizing only for the former is insufficient for a truly immersive XR experience.
  • C2D Latency is Crucial: The introduction of C2D latency as a distinct metric is a key contribution, providing a quantifiable measure for the impact of body motion on perceived latency.
  • Scheduling is Key to Co-optimization: The core challenge lies in co-optimizing latencies for different motion types. BOXR's contention-preventive scheduling policy is central to achieving this by managing task dependencies and resource contention.
  • Runtime Adaptations are Essential: Static optimizations are insufficient due to the dynamic nature of XR. MVIO and SFR demonstrate the power of runtime adaptations to handle motion dynamics and scene changes effectively.
  • Trade-offs are Manageable: The paper shows that by intelligently managing trade-offs (e.g., rendering quality vs. time in SFR, pose quality vs. execution time in MVIO), significant latency reductions can be achieved without severely impacting the user experience.

Key Results

BOXR demonstrates significant improvements across various metrics and platforms:

  • Latency Reduction:
    • In controlled settings, BOXR reduces M2D latency by up to 63% and C2D latency by up to 27%.
  • In practical deployments, BOXR achieves up to 42% reduction in M2D latency and 31% in C2D latency.
  • Frame Rate Increase: BOXR increases frame rate by up to 43% in controlled settings.
  • Miss Rates: BOXR maintains remarkably low miss rates of 1.6% for M2D and 1.0% for C2D requirements in practical deployments.
  • Robustness: BOXR shows improved robustness in dynamic scenarios, with at most a 10% M2D increase and 6% C2D increase during extreme motion and scene changes.
  • Dropped IMU Sensors: BOXR completely eliminates dropped IMU sensor information, unlike baseline methods that can suffer from significant drops (up to 73%).
  • Rendering Quality: While there's a slight decrease in SSIM (up to 5.7%), it's traded for a substantial decrease in rendering time (up to 30.4%).

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while BOXR significantly improves latency, some minor M2D and C2D miss rates can still occur in extremely challenging scenarios (e.g., sudden motion spikes or large scene swaps), though these are significantly reduced compared to baselines.
  • The current framework is software-level optimization; integrating hardware co-optimization could further enhance performance.
  • Future Work:
    • Investigate software-hardware co-optimization for even greater performance gains.
  • Extend the evaluation to GPU-intensive gaming scenarios, as the current benchmark primarily focuses on data-intensive daily working scenarios.

Summary updated

Abstract

The emergence of standalone XR systems has enhanced user mobility, accommodating both subtle, frequent head motions and substantial, less frequent body motions. However, the pervasively used M2D latency metric, which measures the delay between the most recent motion and its corresponding display update, only accounts for head motions. This oversight can leave users prone to motion sickness if significant body motion is involved. Although existing methods optimize M2D latency through asynchronous task scheduling and reprojection methods, they introduce challenges like resource contention between tasks and outdated pose data. These challenges are further complicated by user motion dynamics and scene changes during runtime. To address these issues, we for the first time introduce the C2D latency metric, which captures the delay caused by body motions, and present BOXR, a framework designed to co-optimize both body and head motion delays within an XR system. BOXR enhances the coordination between M2D and C2D latencies by efficiently scheduling tasks to avoid contentions while maintaining an up-to-date pose in the output frame. Moreover, BOXR incorporates a motion-driven visual inertial odometer to adjust to user motion dynamics and employs scene-dependent foveated rendering to manage changes in the scene effectively. Our evaluations show that BOXR significantly outperforms state-of-the-art solutions in 11 EuRoC MAV datasets across 4 XR applications across 3 hardware platforms. In controlled motion and scene settings, BOXR reduces M2D and C2D latencies by up to 63% and 27%, respectively and increases frame rate by up to 43%. In practical deployments, BOXR achieves substantial reductions in real-world scenarios up to 42% in M2D latency and 31% in C2D latency while maintaining remarkably low miss rates of only 1.6% for M2D requirements and 1.0% for C2D requirements.
BibTeX
@inproceedings{liu2024boxr,
  author = {Cong Liu and Ziliang Zhang and Zexin Li and Hyoseung Kim and Cong Liu},
  title = {{BOXR: Body and head motion Optimization framework for eXtended Reality}},
  booktitle = {Rtss},
  year = {2024},
  doi = {10.1109/rtss62706.2024.00016},
}
Figure