TL;DR
Researchers from XPENG Motors have uncovered the "scaling laws" for stereo matching, revealing that while in-domain performance follows a predictable power law before saturating, cross-domain generalization is far less predictable. Their study introduces XStereo, a high-fidelity synthetic dataset, and demonstrates that for real-world performance, the diversity of training data is significantly more important than simply increasing the number of samples.
What Problem Does This Paper Address?
In the world of deep learning, "scaling laws" (like the famous Chinchilla law for LLMs) provide a roadmap for how much data and compute are needed to reach a certain level of performance. While these laws are well-understood for language and 2D image classification, they remain a mystery for stereo matching—the task of estimating 3D depth by comparing two side-by-side images.
Stereo matching relies heavily on synthetic data because obtaining high-quality ground truth depth in the real world is incredibly difficult and expensive. As researchers move toward "foundation models" trained on millions of synthetic images, a critical question arises: Does more synthetic data always lead to better real-world performance? This paper investigates the relationship between data scale, in-domain accuracy, and zero-shot generalization to natural environments.
Methods
The authors established a rigorous experimental methodology to study data scaling:
- Fixed Architecture: They used a consistent 11M-parameter model based on a CNN backbone with a Gate Recurrent Unit (GRU) update block for iterative refinement.
- Systematic Scaling: They created "nested" sub-datasets, starting from small samples (e.g., 2,500 pairs) and doubling them until reaching the full dataset size.
- Modeling the Law: Instead of using the standard LLM scaling formulas which didn't fit well, they proposed a piecewise scaling function. This function captures two distinct regimes: a power-law regime (where more data helps significantly) and a saturation regime (where performance plateaus).
Datasets
The study utilizes several large-scale synthetic datasets:
- XStereo (New): A physically realistic dataset created by the authors using Unreal Engine 5, featuring 200,000 samples across 162 diverse indoor and outdoor scenarios with randomized lighting and weather.
- Public Benchmarks: Scene Flow, FSD (FoundationStereo Dataset), TartanAir, and FallingThings.
- Evaluation Sets: For cross-domain testing, they used real-world datasets like KITTI12/15, RSRD (road surfaces), and WHUStereo (satellite imagery).
Key Insights and Ideas
- The Saturation Frontier: In-domain performance (testing on the same type of synthetic data used for training) follows a clear power law until it hits a "saturation frontier" ($D_c$). Beyond this point, adding more data provides diminishing returns. Crucially, this frontier appears to be a property of the model architecture itself, remaining constant even if the total dataset size changes.
- Generalization is Not a Power Law: Unlike in-domain performance, cross-domain generalization (e.g., training on synthetic data and testing on real-world KITTI images) does not follow a predictable power law. Performance often plateaus much earlier, and sometimes even degrades slightly with excessive in-domain fitting.
- Diversity > Volume: The "domain gap" between synthetic and real data is the primary bottleneck. The researchers found that increasing the diversity and coverage of the training distribution is far more effective for real-world generalization than simply scaling up the number of samples within a single distribution.
- Effective Fitting Utilization (EFU): The authors introduced EFU as a metric to measure data redundancy. It helps identify how much of a dataset is actually "useful" for a specific model before hitting saturation, providing a guide for more efficient data curation.
Key Results
- Scaling Coefficients: The data scaling coefficients ($\beta$) for stereo matching were found to be in the range of 0.2 to 0.4, which is remarkably similar to the scaling behavior observed in Large Language Models.
- XStereo Quality: The new XStereo dataset proved highly effective, achieving generalization performance on par with or better than much larger, less realistic datasets like FSD.
- Redundancy: The study revealed high levels of redundancy in existing datasets. For example, in the TartanAir dataset, only about 25% of the data was needed to reach the saturation point for their model.
Limitations and Future Work
- Model Scaling: While the paper focuses on data scaling, preliminary results on scaling model capacity (up to 40M parameters) were inconclusive. The authors note that current multi-stage stereo architectures are difficult to scale compared to unified Transformers.
- Hyper-parameter Sensitivity: The study used fixed hyper-parameters across all scales. The authors acknowledge that optimal configurations might change as models and datasets grow.
- Synthetic-to-Real Gap: Even with high-fidelity rendering in XStereo, a significant gap remains. Future work could explore more advanced domain randomization or mixing synthetic data with small amounts of real-world data to further push the generalization frontier.
Summary updated