Stereo depth networks are compared on benchmarks such as KITTI, Middlebury and ETH3D using one number per dataset, for example the average pixel error or the fraction of badly matched pixels. Each benchmark, however, was captured with a particular physical rig: a particular distance between the two cameras (the baseline), lens focal length, and range of distances to the scene. These choices set how large the pixel shifts between the two images are, and how strongly a one-pixel mistake turns into a depth mistake. Benchmark documentation does not report the resulting spread of shifts and depths. Nobody can therefore tell whether a model that "generalizes zero-shot" from synthetic to real data is overcoming a change in appearance or quietly benefiting from (or being hurt by) a change in geometry.
We propose a computed, per-benchmark structural card. It records the distribution of the rig and scene quantities that analytic error models identify as decisive. We pair it with an evaluation protocol that uses the card to split each cross-dataset accuracy drop into a part explained by structural mismatch and a part that remains. The card is only worth anything if it predicts error, so the central experiment is a falsification test. If reweighting or rescaling the test data to match the training geometry does not shrink the generalization gap for current models, then structural parameters are not the neglected variable, and the card reduces to bookkeeping.
Core Claim
Two research threads point at the same blind spot from opposite sides. Physical metrology work shows that rig geometry produces systematic error that later software cannot fully remove. Yang et al. model error as a function of the baseline-to-distance ratio $K = B/Z$ and the angle of the camera axes. They report that error is dominated by object distance when $K<1$ and best controlled for $K \in [1, 1.5]$. Sha et al. show that error is non-uniform across the image: it is smallest at the image centre (0.012 mm in their central region) and grows toward the edges, with the vertical field of view mattering more than the horizontal one. Both papers work entirely with hand-built rigs and calibration targets and never touch a learned matcher or a public dataset.
The dataset-transparency literature shows that undocumented pipeline parameters can account for large apparent performance gaps. Fell et al. reimplemented pathology models from their papers alone and saw, for one method, slide accuracy fall from 92% to 54.7%. Datasheets for Datasets supplies a general documentation template, but it has no vocabulary for sensor geometry.
Our claim is narrower than "document more." For rectified stereo benchmarks, the structural quantities a network actually sees can be computed from released calibration and ground truth, and their mismatch between training and test sets explains a measurable share of reported zero-shot gaps. The quantities are the per-pixel disparity in pixels, $d = fB/Z$, the depth $Z$, the ratio $K$, and the pixel's position in the image. The disparity $d$ is the pixel shift between left and right images, with $f$ the focal length in pixels, $B$ the baseline, and $Z$ the depth. Metric depth error additionally depends on rig geometry through first-order triangulation:
$$\delta Z \approx \frac{Z^2}{fB}\,\delta d,$$
where $\delta d$ is the disparity error. This follows from differentiating $Z = fB/d$. Benchmarks report $\delta d$ alone, which hides the $Z^2/(fB)$ factor that differs by orders of magnitude between a driving rig and a close-range indoor rig.
We are also clear about which metrology results do not carry over. The convergence-angle sweet spots (35° in Yang et al.; 40–50° in Sha et al., which disagree with each other) concern verged metrology rigs. After rectification, convergence reaches the network only indirectly, through resampling and the usable field of view. We therefore record convergence and rectification distortion on the card but do not make them part of the central test. Similarly, driving rigs operate far below $K=1$. The DSEC event baseline is 60 cm (DSEC) while scenes extend to tens of metres. The card makes visible that the regime Yang et al. call ill-conditioned is the normal regime for automotive stereo.
Research Program
The first component is the card itself. For each of Scene Flow, KITTI 2012, KITTI 2015, Middlebury 2014, ETH3D and the DSEC RGB stereo pair, we compute the joint distribution of four quantities over ground-truth pixels: disparity in pixels, metric depth, $K$, and normalized image position. We also record the scalar rig parameters $B$, $f$, resolution and field of view, together with the residual vertical misalignment of the rectified pairs, which is measured from the ground truth. Everything is produced by a released script from public calibration files. This departs deliberately from Gebru et al.'s preference for reflection over automation. Structural facts are measurable, so we measure them, and we reserve the hand-written section for things that cannot be computed, such as known calibration drift or how the ground-truth sensor was mounted. The machine-readable layout follows the spirit of Open Datasheets.
The second component is a structure-matched evaluation. Given a model trained on source data $S$ and evaluated on target $T$, we reweight target pixels so that their structural distribution matches the source distribution, using density-ratio weights over the card variables. We then define the structural share of the gap as
$$\rho = \frac{E_T - E_T^{w}}{E_T - E_S},$$
where $E_S$ is in-domain error, $E_T$ is raw target error, and $E_T^{w}$ is target error after structural reweighting. A $\rho$ near zero means geometry explains nothing. A large $\rho$ means part of what has been credited to appearance robustness is really geometry. Reweighting only works where the two distributions overlap, so we report overlap coverage alongside $\rho$ and never extrapolate outside it.
The third component is a set of interventions, because reweighting is correlational. Rescaling a rectified pair by a factor $s$ multiplies disparities by $s$ while changing appearance only through resolution, which is a direct test of disparity-range mismatch. On the synthetic side, we draw subsets of Scene Flow whose disparity and position distributions are matched, or deliberately mismatched, to each real target, while appearance is held fixed. Models trained on structure-matched and unmatched subsets of the same size then isolate structure from appearance. DSEC's account of MVSEC-trained models failing on DSEC confounds a 6x baseline change with a 3x resolution change. Our design separates those two factors.
Baselines
The naive reference point is current practice. A model is trained on Scene Flow, and its unweighted end-point error (average pixel distance between predicted and true disparity) and bad-pixel rates on each target are reported as a single number, with the whole gap attributed implicitly to appearance. Every other result is measured against this. The proposal beats it if the card-based decomposition gives stable, nonzero $\rho$ that holds across models and seeds, and if card variables predict per-image error better than image-statistics descriptors do.
The strongest existing methods are StereoBase from OpenStereo, IGEV-Stereo and RAFT-Stereo. These are the models whose zero-shot claims are at stake. OpenStereo reports StereoBase at 0.34 end-point error on Scene Flow versus 0.47 for IGEV-Stereo, and a KITTI 2015 D1-all of 1.44%. For zero-shot Middlebury and ETH3D the file gives no figures: no reported result found on Lacuna for zero-shot Middlebury or ETH3D scalar error of these models. We will therefore reproduce all three inside the OpenStereo toolbox with its published configurations, the same Scene Flow training data, and the same schedule and augmentation. Our first deliverable is matching the 0.34 and 0.47 Scene Flow figures, so that all later comparisons are on equal footing. The question here is not whether we beat these models. It is whether their ranking and their gaps survive structure matching, and a ranking change under reweighting would count as a substantive finding.
Two ablations isolate the key ingredient. The first reweights by disparity magnitude alone, dropping depth, $K$ and image position. It tests whether the multi-variable card adds anything beyond a disparity histogram. The second reweights with randomly permuted weights that keep the same weight distribution. It controls for the variance that reweighting itself introduces. The full card must beat both to justify itself.
Why not just…?
Why not just stratify existing metrics by ground-truth disparity range? This is the most serious threat, and the disparity-only ablation is built to settle it. If disparity-range stratification captures nearly all of $\rho$, the honest conclusion is that a disparity histogram is the whole card, and we will report that. Depth, $K$ and image position can still contribute in two ways. Sha et al.'s edge-of-image degradation implies an effect of position that disparity alone does not capture. And the metric-depth term $Z^2/(fB)$ determines which disparity errors matter at all.
Why not just use scale and crop augmentation, which already randomizes disparity ranges? Augmentation may already cover much of the structural gap. In that case, models trained with heavier scale augmentation should show smaller $\rho$, and the decomposition will measure that directly. The rescaling intervention also shows what augmentation can and cannot reproduce. Rescaling changes disparity in pixels but cannot change the depth-error amplification fixed by the physical $f$ and $B$, a limitation consistent with OpenStereo's note that its swap-flip augmentation helps real-world transfer but slightly hurts synthetic performance.
Why not just extend Gebru et al.'s datasheet with a few geometry questions? Extra free-text fields would record rig specifications, but they would not tell anyone whether those specifications matter. The case for a card rests on predictive validity. Fell et al. show that domain-specific, checkable items are what make reproduction possible, but they validated their checklist by reimplementation, not by prediction. We hold our card to that higher standard: if card variables do not predict error, we will recommend against it.
Proposed Experiments
The decisive check comes first. For each of the three models and each of the four real targets, we compute $\rho$ with bootstrap intervals over images. The structural hypothesis is falsified for disparity metrics if $\rho$ is indistinguishable from the random-weight control across all model–target pairs. The second check is causal. On Scene Flow subsets matched to KITTI 2015 versus unmatched subsets of equal size, zero-shot D1-all on KITTI 2015 should differ by more than seed variance if structure matters, with appearance held constant by construction. The third check covers metric depth. We convert each model's disparity errors to depth errors with each benchmark's $f$ and $B$ and test whether model rankings in depth differ from rankings in disparity, especially on DSEC, whose wide baseline sits at the opposite end from narrow-rig data. Finally, we break down error by image region on Middlebury and ETH3D to test whether learned matchers show the centre-to-edge degradation Sha et al. measured on physical rigs, including the stronger vertical effect.
The released artifacts are the cards, the reweighting and rescaling protocol as an OpenStereo evaluation module, and the decomposition tables. A positive result would tell the field that part of what is reported as appearance generalization is geometry. A negative result would tell it, with evidence, that rectified deep stereo is largely insensitive to the structural parameters metrology cares about.
References
- System Structural Error Analysis in Binocular Vision Measurement Systems
- The analysis of the structural parameter influences on measurement errors in a binocular 3D reconstruction system: a portable 3D system
- Reproducibility of deep learning in digital pathology whole slide image analysis
- Datasheets for datasets
- DSEC: A Stereo Event Camera Dataset for Driving Scenarios
- Open Datasheets: Machine-readable Documentation for Open Datasets and Responsible AI Assessments
- OpenStereo: A Comprehensive Benchmark for Stereo Matching and Strong Baseline
- Iterative Geometry Encoding Volume for Stereo Matching
- RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching