Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended Reality

Ziang Cheng, Jiayu Yang, Hongdong Li

TL;DR

This paper introduces XR-Stereo, a high-fidelity synthetic dataset for indoor video stereo matching, and a novel framework that achieves over 100 FPS by "unrolling" iterative refinement across time. By leveraging temporal redundancy and 6-DoF camera poses, the authors demonstrate that high-accuracy depth sensing is possible even on low-power mobile XR headsets.

What Problem Does This Paper Address?

Stereo matching—calculating depth from two offset camera views—is a "cornerstone" for Extended Reality (XR). Whether it’s for placing a virtual object on a real table or understanding an indoor environment, XR needs depth data that is both highly accurate and extremely fast.

However, researchers face two major hurdles:

  1. Data Scarcity: Most stereo datasets focus on autonomous driving (like KITTI) or lack the photorealistic textures and complex camera motions (6-DoF) typical of a human wearing a VR/AR headset.
  2. Computational Cost: Modern deep-learning models, such as RAFT-Stereo, achieve great accuracy by iteratively refining their depth estimates. Unfortunately, running 20+ iterations per frame is far too taxing for the battery-powered processors found in standalone headsets like the Meta Quest or HoloLens.

Methods

The authors propose a "Stereo Matching in Time" framework. Instead of treating every video frame as a brand-new problem, they recognize that consecutive frames are highly redundant.

The Core Innovation: Temporal Unrolling

Standard iterative models perform multiple refinement steps (using a Gated Recurrent Unit, or GRU) on a single frame before moving to the next. This paper "unrolls" these iterations across the temporal dimension. They perform just one or two iterations per frame but carry the "hidden state" of the GRU forward to the next frame.

Temporal Warping

To make this work, the model must account for the user's head movement. As shown in the framework overview, they use the HMD’s known camera pose to warp the previous frame's disparity and GRU hidden state into the current frame’s coordinate system.

Real-time video stereo matching framework. The iterative process is distributed across frames to save computation.

This "warm-starts" the refinement process for the new frame, allowing the model to achieve high accuracy with a fraction of the per-frame computation. They use Softmax Splatting to handle occlusions during this warping process, ensuring that foreground objects correctly overwrite background pixels when the viewpoint shifts.

Datasets

The authors introduced the XR-Stereo Dataset to bridge the gap in indoor XR training data.

  • XR-Stereo (New): Contains 60,000 photorealistic stereo pairs across 13 indoor environments (living rooms, offices, etc.). Crucially, the camera paths were recorded from actual human head movements using VR/AR headsets, providing realistic 6-DoF trajectories.
  • KITTI VO: Used to evaluate how well the model generalizes to real-world, outdoor driving scenarios.
The XR-Stereo dataset features photorealistic indoor scenes and realistic 6-DoF camera trajectories.

Key Insights and Ideas

  • Temporal Redundancy is a "Free Lunch": In a 30Hz or 60Hz video stream, the change between frames is minute. Recomputing the entire cost volume and running full iterations from scratch is wasteful.
  • Pose-Awareness is Key: By leveraging the 6-DoF tracking already present in XR headsets, the model can geometrically align features across time, making the temporal "warm-start" significantly more effective than simple frame-to-frame tracking.
  • Efficiency vs. Accuracy: The paper proves that you don't need to choose one. By distributing the workload over time, you can maintain the accuracy of a heavy iterative model while operating at the speed of a lightweight one.

Key Results

The quantitative improvements are substantial, particularly regarding the trade-off between error (EPE) and inference time.

  • Speed: The "Ours-Fast" model achieves a blistering 134 FPS on a desktop GPU (RTX 3090 Ti) and a steady 30 FPS on a mobile Qualcomm XR2 chip.
  • Accuracy: On the XR-Stereo dataset, their 1-iteration temporal model outperformed the 20-iteration RAFT-Stereo baseline while being over 5x faster.
  • Robustness: As seen in the qualitative results, the model maintains sharp edges and handles textureless surfaces (like white walls) better than previous real-time baselines.
Comparison of EPE vs. Inference Time. The proposed models (blue/green) sit significantly lower and further left than the baseline (red).
Qualitative results showing the estimated disparity and error maps. The proposed method produces much cleaner depth maps than the baseline.

Limitations and Future Work

While a major step forward, the authors note a few areas for improvement:

  • Static Scenes: The current XR-Stereo dataset does not include moving objects (like pets or people), which are common in real-world XR.
  • Initialization: Because the model relies on temporal history, the first few frames of a video (roughly the first 0.5 seconds) are less accurate as the GRU state "warms up."
  • Egocentric Models: The dataset lacks virtual hands or bodies, which are essential for many AR interactions.

Future work will likely focus on extending this temporal framework to scene flow estimation, which tracks the 3D motion of every pixel in the scene.

Summary updated

Abstract

Real-time Stereo Matching is a cornerstone algorithm for many Extended Reality (XR) applications, such as indoor 3D understanding, video pass-through, and mixed-reality games. Despite significant advancements in deep stereo methods, achieving real-time depth inference with high accuracy on a low-power device remains a major challenge. One of the major difficulties is the lack of high-quality indoor video stereo training datasets captured by head-mounted VR/AR glasses. To address this issue, we introduce a novel video stereo synthetic dataset that comprises photorealistic renderings of various indoor scenes and realistic camera motion captured by a 6-DoF moving VR/AR head-mounted display (HMD). This facilitates the evaluation of existing approaches and promotes further research on indoor augmented reality scenarios. Our newly proposed dataset enables us to develop a novel framework for continuous video-rate stereo matching. As another contribution, our dataset enables us to proposed a new video-based stereo matching approach tailored for XR applications, which achieves real-time inference at an impressive 134fps on a standard desktop computer, or 30fps on a battery-powered HMD. Our key insight is that disparity and contextual information are highly correlated and redundant between consecutive stereo frames. By unrolling an iterative cost aggregation in time (i.e. in the temporal dimension), we are able to distribute and reuse the aggregated features over time. This approach leads to a substantial reduction in computation without sacrificing accuracy. We conducted extensive evaluations and comparisons and demonstrated that our method achieves superior performance compared to the current state-of-the-art, making it a strong contender for real-time stereo matching in VR/AR applications.
BibTeX
@inproceedings{cheng2024stereo,
  author = {Ziang Cheng and Jiayu Yang and Hongdong Li},
  title = {{Stereo Matching in Time: 100+ FPS Video Stereo Matching for Extended Reality}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00852},
}
Figure