TL;DR
This paper introduces XR-Stereo, a high-fidelity synthetic dataset for indoor video stereo matching, and a novel framework that achieves over 100 FPS by "unrolling" iterative refinement across time. By leveraging temporal redundancy and 6-DoF camera poses, the authors demonstrate that high-accuracy depth sensing is possible even on low-power mobile XR headsets.
What Problem Does This Paper Address?
Stereo matching—calculating depth from two offset camera views—is a "cornerstone" for Extended Reality (XR). Whether it’s for placing a virtual object on a real table or understanding an indoor environment, XR needs depth data that is both highly accurate and extremely fast.
However, researchers face two major hurdles:
- Data Scarcity: Most stereo datasets focus on autonomous driving (like KITTI) or lack the photorealistic textures and complex camera motions (6-DoF) typical of a human wearing a VR/AR headset.
- Computational Cost: Modern deep-learning models, such as RAFT-Stereo, achieve great accuracy by iteratively refining their depth estimates. Unfortunately, running 20+ iterations per frame is far too taxing for the battery-powered processors found in standalone headsets like the Meta Quest or HoloLens.
Methods
The authors propose a "Stereo Matching in Time" framework. Instead of treating every video frame as a brand-new problem, they recognize that consecutive frames are highly redundant.
The Core Innovation: Temporal Unrolling
Standard iterative models perform multiple refinement steps (using a Gated Recurrent Unit, or GRU) on a single frame before moving to the next. This paper "unrolls" these iterations across the temporal dimension. They perform just one or two iterations per frame but carry the "hidden state" of the GRU forward to the next frame.
Temporal Warping
To make this work, the model must account for the user's head movement. As shown in the framework overview, they use the HMD’s known camera pose to warp the previous frame's disparity and GRU hidden state into the current frame’s coordinate system.
This "warm-starts" the refinement process for the new frame, allowing the model to achieve high accuracy with a fraction of the per-frame computation. They use Softmax Splatting to handle occlusions during this warping process, ensuring that foreground objects correctly overwrite background pixels when the viewpoint shifts.
Datasets
The authors introduced the XR-Stereo Dataset to bridge the gap in indoor XR training data.
- XR-Stereo (New): Contains 60,000 photorealistic stereo pairs across 13 indoor environments (living rooms, offices, etc.). Crucially, the camera paths were recorded from actual human head movements using VR/AR headsets, providing realistic 6-DoF trajectories.
- KITTI VO: Used to evaluate how well the model generalizes to real-world, outdoor driving scenarios.
Key Insights and Ideas
- Temporal Redundancy is a "Free Lunch": In a 30Hz or 60Hz video stream, the change between frames is minute. Recomputing the entire cost volume and running full iterations from scratch is wasteful.
- Pose-Awareness is Key: By leveraging the 6-DoF tracking already present in XR headsets, the model can geometrically align features across time, making the temporal "warm-start" significantly more effective than simple frame-to-frame tracking.
- Efficiency vs. Accuracy: The paper proves that you don't need to choose one. By distributing the workload over time, you can maintain the accuracy of a heavy iterative model while operating at the speed of a lightweight one.
Key Results
The quantitative improvements are substantial, particularly regarding the trade-off between error (EPE) and inference time.
- Speed: The "Ours-Fast" model achieves a blistering 134 FPS on a desktop GPU (RTX 3090 Ti) and a steady 30 FPS on a mobile Qualcomm XR2 chip.
- Accuracy: On the XR-Stereo dataset, their 1-iteration temporal model outperformed the 20-iteration RAFT-Stereo baseline while being over 5x faster.
- Robustness: As seen in the qualitative results, the model maintains sharp edges and handles textureless surfaces (like white walls) better than previous real-time baselines.
Limitations and Future Work
While a major step forward, the authors note a few areas for improvement:
- Static Scenes: The current XR-Stereo dataset does not include moving objects (like pets or people), which are common in real-world XR.
- Initialization: Because the model relies on temporal history, the first few frames of a video (roughly the first 0.5 seconds) are less accurate as the GRU state "warms up."
- Egocentric Models: The dataset lacks virtual hands or bodies, which are essential for many AR interactions.
Future work will likely focus on extending this temporal framework to scene flow estimation, which tracks the 3D motion of every pixel in the scene.
Summary updated