Bridging Stereo Matching and Optical Flow via Spatiotemporal Correspondence

Hsueh-Ying Lai, Yi-Hsuan Tsai, Wei-Chen Chiu

Introduction

To achieve a comprehensive understanding of 3D scenes in motion—critical for applications like autonomous driving and robotics—vision systems must simultaneously determine object depth and motion. Traditionally, computer vision treats these as two independent tasks: stereo matching, which calculates depth from horizontal pixel displacement between left and right camera views, and optical flow, which tracks 2D pixel motion across consecutive video frames.

This paper identifies a significant inefficiency in this decoupled approach: most existing research utilizes separate networks for each task, failing to exploit shared feature representations or the inherent geometric relationships between space and time. Furthermore, obtaining ground-truth labels for these tasks is prohibitively expensive. To address this, the authors propose a unified, unsupervised network that jointly learns both tasks by enforcing spatiotemporal consistency.

Methodology

The core contribution is a single, principled network architecture designed to handle both stereo and temporal image pairs. Rather than treating them as isolated problems, the framework views them as different perspectives of a single spatiotemporal puzzle.

Spatiotemporal Correspondence

The model processes "temporally adjacent stereo pairs" (Left and Right images at times $t$ and $t+1$). As illustrated in Figure 1, the network estimates disparity maps ($D$) for spatial correspondence and flow maps ($F$) for temporal correspondence, forming a closed loop of reconstructions.

Spatiotemporal relationships between stereo pairs across time.

The 2-Warp Consistency Loss

The "bridge" between these tasks is the novel 2-Warp consistency loss. This geometric constraint posits that if an image is warped twice—for example, moving from the Left camera at time $t+1$ to the Left camera at time $t$ (flow), and then to the Right camera at time $t$ (stereo)—the resulting image should match the actual Right camera image at time $t$. This self-checking mechanism provides a powerful supervisory signal without requiring human-labeled data.

To ensure the model learns from reliable data, the authors implement occlusion-aware learning. A forward-backward consistency check is used to mask out pixels that disappear behind objects or move off-screen, preventing the loss function from being corrupted by occluded regions.

Experimental Results

The researchers evaluated the method on the KITTI benchmark, using the KITTI Raw dataset for large-scale unsupervised training. The results demonstrate that joint learning provides mutual benefits: the shared features become more robust, and the "easier" stereo task helps regularize the learning process for the more complex optical flow task.

Quantitative and Qualitative Performance

The "Full Model" consistently outperformed previous unsupervised state-of-the-art methods:

  • Depth Estimation: On the KITTI 2015 stereo set, the model achieved an Absolute Relative (Abs Rel) error of 0.0620, a significant improvement over prior unsupervised baselines.
  • Optical Flow: On KITTI 2012, the model reached an Average End-Point Error (EPE) of 2.56, outperforming specialized flow networks like UnFlow-CSS (3.29).

Qualitatively, the joint model produces sharper object boundaries and handles thin structures, such as poles and distant vehicles, with much higher fidelity than models trained on a single task alone.

Qualitative comparison of depth and flow maps against ground truth.

Conclusion

By bridging stereo matching and optical flow through a unified network and geometric consistency, this research shows that spatial and temporal tasks are deeply interconnected. While the model currently performs best on static backgrounds, it represents a significant step toward complete, unsupervised 3D scene understanding. Future work may involve integrating camera egomotion more tightly into the framework to further enhance performance in complex, dynamic environments.

Summary updated

BibTeX
@inproceedings{lai2019bridging,
  author = {Hsueh-Ying Lai and Yi-Hsuan Tsai and Wei-Chen Chiu},
  title = {{Bridging Stereo Matching and Optical Flow via Spatiotemporal Correspondence}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=Dz2HnrrEfAB},
}
Figure