Data Scaling Laws in Stereo Matching: From In-domain Fitting to Cross-domain Generalization

Tong Zhao, Di Liang, Yiming Zeng, Bo Xu, Yingzhi Tang, Chen BaiarXiv 2026 · arXiv

TL;DR

Researchers from XPENG Motors have uncovered the "scaling laws" for stereo matching, revealing that while in-domain performance follows a predictable power law before saturating, cross-domain generalization is far less predictable. Their study introduces XStereo, a high-fidelity synthetic dataset, and demonstrates that for real-world performance, the diversity of training data is significantly more important than simply increasing the number of samples.

What Problem Does This Paper Address?

In the world of deep learning, "scaling laws" (like the famous Chinchilla law for LLMs) provide a roadmap for how much data and compute are needed to reach a certain level of performance. While these laws are well-understood for language and 2D image classification, they remain a mystery for stereo matching—the task of estimating 3D depth by comparing two side-by-side images.

Stereo matching relies heavily on synthetic data because obtaining high-quality ground truth depth in the real world is incredibly difficult and expensive. As researchers move toward "foundation models" trained on millions of synthetic images, a critical question arises: Does more synthetic data always lead to better real-world performance? This paper investigates the relationship between data scale, in-domain accuracy, and zero-shot generalization to natural environments.

Methods

The authors established a rigorous experimental methodology to study data scaling:

  1. Fixed Architecture: They used a consistent 11M-parameter model based on a CNN backbone with a Gate Recurrent Unit (GRU) update block for iterative refinement.
  2. Systematic Scaling: They created "nested" sub-datasets, starting from small samples (e.g., 2,500 pairs) and doubling them until reaching the full dataset size.
  3. Modeling the Law: Instead of using the standard LLM scaling formulas which didn't fit well, they proposed a piecewise scaling function. This function captures two distinct regimes: a power-law regime (where more data helps significantly) and a saturation regime (where performance plateaus).
Figure 2. The general framework of deep stereo matching models.

Datasets

The study utilizes several large-scale synthetic datasets:

  • XStereo (New): A physically realistic dataset created by the authors using Unreal Engine 5, featuring 200,000 samples across 162 diverse indoor and outdoor scenarios with randomized lighting and weather.
  • Public Benchmarks: Scene Flow, FSD (FoundationStereo Dataset), TartanAir, and FallingThings.
  • Evaluation Sets: For cross-domain testing, they used real-world datasets like KITTI12/15, RSRD (road surfaces), and WHUStereo (satellite imagery).

Key Insights and Ideas

  • The Saturation Frontier: In-domain performance (testing on the same type of synthetic data used for training) follows a clear power law until it hits a "saturation frontier" ($D_c$). Beyond this point, adding more data provides diminishing returns. Crucially, this frontier appears to be a property of the model architecture itself, remaining constant even if the total dataset size changes.
  • Generalization is Not a Power Law: Unlike in-domain performance, cross-domain generalization (e.g., training on synthetic data and testing on real-world KITTI images) does not follow a predictable power law. Performance often plateaus much earlier, and sometimes even degrades slightly with excessive in-domain fitting.
  • Diversity > Volume: The "domain gap" between synthetic and real data is the primary bottleneck. The researchers found that increasing the diversity and coverage of the training distribution is far more effective for real-world generalization than simply scaling up the number of samples within a single distribution.
  • Effective Fitting Utilization (EFU): The authors introduced EFU as a metric to measure data redundancy. It helps identify how much of a dataset is actually "useful" for a specific model before hitting saturation, providing a guide for more efficient data curation.
Left: In-domain error shows a clear power law followed by saturation. Right: Cross-domain error on KITTI12 is less predictable and plateaus early.

Key Results

  • Scaling Coefficients: The data scaling coefficients ($\beta$) for stereo matching were found to be in the range of 0.2 to 0.4, which is remarkably similar to the scaling behavior observed in Large Language Models.
  • XStereo Quality: The new XStereo dataset proved highly effective, achieving generalization performance on par with or better than much larger, less realistic datasets like FSD.
  • Redundancy: The study revealed high levels of redundancy in existing datasets. For example, in the TartanAir dataset, only about 25% of the data was needed to reach the saturation point for their model.

Limitations and Future Work

  • Model Scaling: While the paper focuses on data scaling, preliminary results on scaling model capacity (up to 40M parameters) were inconclusive. The authors note that current multi-stage stereo architectures are difficult to scale compared to unified Transformers.
  • Hyper-parameter Sensitivity: The study used fixed hyper-parameters across all scales. The authors acknowledge that optimal configurations might change as models and datasets grow.
  • Synthetic-to-Real Gap: Even with high-fidelity rendering in XStereo, a significant gap remains. Future work could explore more advanced domain randomization or mixing synthetic data with small amounts of real-world data to further push the generalization frontier.

Summary updated

Abstract

In the era of large models, stereo matching is developing towards generalizable foundation models trained on million-scale synthetic images. However, how model performance scales up with synthetic data is still underexplored. In this work, we preliminarily investigate the in-domain and cross-domain data scaling laws in stereo matching. A realistic synthetic dataset, named XStereo, containing 200 thousand samples is first built to serve as a high-quality benchmark for data scaling research. In-domain training on five large-scale datasets demonstrate a two-regime scaling behavior with data, transitioning from the standard power law to saturation, where the critical transition point is approximately invariant under a fixed model. Cross-domain performance exhibits a substantially weaker and less predictable dependence on the source-domain data scale. The distribution and coverage diversity are more effective for diminishing domain gaps. More instructive suggestions are provided for high-efficiency data construction and training. The insightful findings are significant verification and extension of the existing scaling laws in both language and vision, paving the way to efficient and stronger stereo matching. The XStereo dataset is public at https://huggingface.co/datasets/Tong-Zhao/XStereo.
BibTeX
@misc{zhao2026data,
  author = {Tong Zhao and Di Liang and Yiming Zeng and Bo Xu and Yingzhi Tang and Chen Bai},
  title = {{Data Scaling Laws in Stereo Matching: From In-domain Fitting to Cross-domain Generalization}},
  year = {2026},
  eprint = {2610.38600},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38600},
}
Figure