Most learned multi-view depth systems estimate depth the same way. They slice the space in front of the camera into a stack of parallel depth planes. They score how well the images agree at each plane, smooth those scores with a 3D network that looks at neighbouring pixels and neighbouring planes, and then read off one depth per pixel. The smoothing over neighbours (the volumetric context) is what makes these systems work on blank walls and repeated patterns. The plane stack is what makes slanted floors come out as staircases. The usual final step adds a second artifact: it takes the probability-weighted average depth, which at an object edge lands halfway between foreground and background, on a surface that does not exist.
This proposal keeps the 3D smoothing and changes two things that happen around it. First, the per-pixel scores become a continuous distribution over depth, built from a small offset per plane plus a Laplacian kernel. Depth is read out as the median of that distribution, not the mean. Second, in a coarse-to-fine cascade, the next stage's search range is set from quantiles of that distribution: it is centred on the median and its width follows where the probability mass actually lies. The point is that the stage-to-stage handoff, not only the final readout, is where discretization errors become permanent. We also introduce a measurement that turns "staircase effect" into a number: the part of the depth error that depends on where the true depth falls between two adjacent planes. This lets us tell how much of the artifact comes from the readout and how much from the 3D network itself.
Core Claim
Continuous median readouts have been developed only for two-camera stereo on rectified image pairs. Stereo Risk models disparity as a continuous density and finds its median by binary search, with gradients through the implicit function theorem. PCVNet parameterizes disparity as a four-component Gaussian mixture. Neither touches multi-view frustum volumes or cascades. Multi-view systems such as MVSNet, CasMVSNet, and MVSFormer++ keep the weighted-average or classification readout. CasMVSNet passes that estimate to the next stage with a fixed, shrinking interval, which is exactly the error-propagation weakness the research file describes.
We claim three things. A median readout inside a cascade reduces ghost depths at occlusion edges. Quantile-derived intervals let later stages recover from coarse-stage mistakes that fixed intervals lock in. Together these shrink the between-planes error signature on slanted surfaces without adding planes or memory.
What is new is the cascade handoff and the diagnostic. Porting the readout alone would be incremental, so we treat it only as one ablation arm.
The Mechanism
At each stage the 3D network outputs logits over $D$ planes at depths $d_i$, as in the original systems. We add a per-plane offset head, in the spirit of the continuous offsets of Wasserstein Distances for Stereo Disparity Estimation. This defines a per-pixel density $p(d)=\sum_i w_i\,\mathrm{Laplace}(d;\,d_i+o_i,\,b)$. Here $w_i$ are the softmax weights, $o_i$ the predicted offsets, and $b$ a learned per-pixel scale.
Let $F$ be the cumulative distribution of $p$. The depth estimate $\hat d$ solves $F(\hat d)=0.5$, found by binary search as in Stereo Risk. Its gradient with respect to network parameters $\theta$ follows from the implicit function theorem:
$$\frac{\partial \hat d}{\partial \theta} = -\frac{\partial F(\hat d)/\partial \theta}{p(\hat d)}.$$
The next stage samples its planes uniformly in $[F^{-1}(q),\,F^{-1}(1-q)]$, with a floor on the width. The tail level $q$ is fixed (e.g. 0.05) or learned per stage. A sharply peaked distribution therefore gets a narrow, fine-grained next stage. A two-peaked distribution at an occlusion edge gets a range wide enough to cover both peaks, instead of a narrow window centred between them.
Training uses an $L^1$ loss on the median at every stage. We add a calibration term, the negative log-likelihood of ground truth under $p$, so that the quantiles mean something. Memory is essentially unchanged: one extra channel per plane, plus a binary search with tens of iterations per pixel.
The Staircase Diagnostic
Existing benchmarks report overall accuracy and completeness. These numbers mix quantization error with matching error.
For each valid pixel, let $r=\hat d-d^*$ be the depth error and $\phi$ the fractional position of the true depth $d^*$ between its two neighbouring planes in the final stage, so $\phi\in[0,1)$. A perfectly continuous estimator has $\mathbb{E}[r\mid\phi]$ flat in $\phi$. A staircase estimator produces a sawtooth whose peak-to-peak size approaches the plane spacing, because each plateau is pulled toward the nearest plane. We define the staircase amplitude as
$$A=\max_\phi \mathbb{E}[r\mid\phi]-\min_\phi \mathbb{E}[r\mid\phi],$$
reported relative to the plane spacing and restricted to pixels whose ground-truth surface is slanted beyond a threshold angle to the camera.
We complement $A$ with surface-normal error from ground-truth meshes, since staircases show up directly as wrong normals. We also report the error at pixels near ground-truth depth discontinuities, to measure ghosting.
The diagnostic separates two hypotheses. If the median readout drives $A$ toward zero, the readout was responsible. If $A$ stays large, the 3D network has learned the plateau bias that Semi-Global Stereo Matching with Surface Orientation Priors attributes to label-smoothness objectives. That finding would itself be publishable, and it would redirect follow-up work toward the regularizer.
Research Program
We implement the readout and quantile handoff as a drop-in module for two backbones. The first is CasMVSNet, the simplest cascade. The second is MVSFormer++, the strongest multi-view system in the file, whose cost-volume transformer is described as fully tied to discrete planes.
Training follows each backbone's published recipe on DTU, with BlendedMVS fine-tuning where the original did so. We then evaluate on DTU, the Tanks and Temples intermediate and advanced sets, and ETH3D high-resolution scenes. ETH3D and DTU are the benchmarks whose authors (ETH3D benchmark, DTU evaluation) single out thin structures and oblique surfaces as the main failure cases.
We compute the staircase diagnostic on DTU and ETH3D from ground truth, at matched plane counts. As a controlled check, we also run every method on a 2D stereo backbone on SceneFlow, where Stereo Risk's own numbers give a sanity reference.
Baselines
The naive reference point is each backbone exactly as published, with its own readout and fixed interval schedule. For CasMVSNet the file reports a DTU overall score of 0.355 mm, as cited in RayMVSNet++ and TransMVSNet. We rerun it with identical data, views, resolution and training epochs, so that gains come only from the module.
The strongest existing method is MVSFormer++. The file reports a DTU overall error of 0.2805 mm and Tanks and Temples mean F-scores of 67.03 (intermediate) and 41.70 (advanced) for MVSFormer++. We retrain it with its own recipe under the same compute budget, with and without our module, and use the same hyperparameter-search budget for both arms. Beating it means three things together: a lower DTU overall error than the retrained baseline over three seeds, no loss in Tanks and Temples F-score, and a clearly smaller staircase amplitude and edge error. If $A$ drops but overall error does not, we will report that honestly as an artifact fix without an accuracy gain.
We also compare against Effi-MVS, which reports 0.317 mm on DTU. It is the strongest competitor that also centres a narrow window on the current estimate, so it tests whether iterative recentring alone does the job.
Three ablations isolate the key ingredient. The first applies the median readout with CasMVSNet's fixed intervals, which amounts to porting Stereo Risk. The second uses the mean readout with quantile intervals. The third uses intervals derived from the standard deviation of the discrete distribution instead of quantiles. The third is the decisive one: if standard-deviation intervals match quantile intervals, the median-based handoff adds nothing beyond simple uncertainty-sized windows.
Why not just…?
Why not just add more depth planes and keep the weighted average? Finer planes do shrink the sawtooth amplitude, but memory grows linearly in plane count on top of the image grid, and MVSNet already had to downsample inputs to fit. Finer planes also do nothing for the averaging between two peaks that Stereo Risk identifies as the cause of ghost depths. We test this directly by doubling and quadrupling planes in the baseline. We report the plane count at which its staircase amplitude matches ours, and the memory that costs. If a modest increase closes both the edge error and $A$, our claim fails.
Why not just take the argmax plane and average only within a small window around it? This is the most serious threat. It removes ghosting at edges cheaply and is common practice. It still leaves the readout tied to plane positions, and it gives the next cascade stage no principled range. We include it as a full arm, with and without fixed intervals. Our method must beat it specifically on the coarse-stage recovery measurement, defined in the experiments below. If it ties there, the quantile handoff is not justified, and we would say so.
Why not just refine the depth map with a 2D network guided by the image? MVSNet already includes such a refinement network and still sits at about 0.355 mm on DTU with oversmoothed boundaries. A 2D filter also cannot consult the other views. We nonetheless train an RGB-guided refinement network on top of the baseline with matched parameter count and report its staircase amplitude. This shows whether $A$ is something image-space filtering can remove.
Why not just use a Gaussian mixture as in PCVNet? A mixture with a few components also represents two peaks. Its mean, though, has the same ghosting problem, and the file gives no evidence of it working in multi-view cascades. We run a mixture-readout arm with mean and with median, which separates the choice of distribution family from the choice of statistic.
Proposed Experiments and Decisive Checks
The first decisive check targets the handoff. On DTU and ETH3D we take the pixels where the coarse stage's estimate is wrong by more than the fixed next-stage interval. We then measure what fraction end within tolerance after the final stage, under fixed, standard-deviation and quantile intervals. The hypothesis is that quantile handoff raises this recovery rate substantially, and that it does so mainly on two-peaked pixels near edges. This is where the proposal stands or falls.
The second check is the staircase amplitude on slanted surfaces, at matched plane count and memory. Each arm is reported against the plane-doubling curve.
The third check is end-to-end accuracy: DTU overall error and Tanks and Temples F-score against the retrained MVSFormer++. We also report memory, runtime and the cost of the binary search.
Finally, on SceneFlow we confirm that the readout reproduces the advantage Stereo Risk reports (an end-point error of 0.43 against 0.47 for IGEV). This guards against implementation error before attributing any multi-view result to the method.
References
- Stereo Risk: A Continuous Modeling Approach to Stereo Matching
- Parameterized Cost Volume for Stereo Matching
- MVSNet: Depth Inference for Unstructured Multi-view Stereo
- Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching
- MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo
- Wasserstein Distances for Stereo Disparity Estimation
- Semi-global Stereo Matching with Surface Orientation Priors
- A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos
- Large Scale Multi-view Stereopsis Evaluation
- Efficient Multi-view Stereo by Iterative Dynamic Cost Volume