Training Coarse Stages to Hand Off the Truth: A Coverage Loss for Uncertainty-Guided Search

Many depth and stereo networks search in stages. A coarse stage guesses roughly where the answer lies, and a fine stage searches only a small window around that guess. Uncertainty-guided methods size the window with the coarse stage's confidence: narrow where it is sure, wide where it is not. Every one of these systems, however, trains the coarse stage with the same loss used for the final answer, which rewards a sharp, confident guess. That is the wrong job description. The coarse stage does not need to be right. It needs to hand the fine stage a small set of candidate depths that contains the truth. When the coarse stage commits sharply to the wrong surface, as it often does at object boundaries where foreground and background both look plausible, the truth is pruned and no later stage can recover it.

We propose to train the coarse stage directly for this handoff. The new loss asks whether at least one of the next stage's candidate positions falls within that stage's reach of the ground truth. It rewards placing candidates on both plausible surfaces and penalizes spending a fixed candidate budget on regions that cannot contain the answer. This is a surrogate-loss misalignment problem located inside an adaptive sampling pipeline. The evaluation quantity for the intermediate stage is a step function (covered or not), while the training signal is a regression loss on a different quantity.

Core Claim

Cascade and iterative depth estimators share one structural misalignment. The intermediate distribution is used at inference to place hypotheses, but during training it is optimized to make an accurate point estimate. Point-estimate losses reward collapsing probability mass onto one mode. Hypothesis placement needs mass kept on every mode that might be true, sized to the downstream search budget. We claim that a differentiable set-coverage loss, applied to the hypotheses actually generated from the intermediate distribution, reduces irreversible coarse-stage failures. The reductions should concentrate at depth discontinuities and under domain shift, and should hold at equal or lower hypothesis counts. We further claim that these gains cannot be reproduced by single-interval coverage methods such as quantile-regressed bounds with post-hoc calibration, because a single interval at a bimodal boundary pixel must either span the empty gap between surfaces or drop one of them.

What the Two Directions Contain, and Where This Sits

On the sampling side, several methods use the variance of the intermediate distribution to size search windows. CFNet does this across a coarse-to-fine cascade. Adaptive Thin Volumes does it for multi-view stereo. IEBins widens a single target bin by its variance before subdividing it. These works motivate the proposal: they show that early errors lock in, and that elastic windows help. They also constrain it, because a variance-sized single window is exactly the design that fails on bimodal pixels. UASNet is the closest work. It notes that variance cannot distinguish one diffuse mode from two separated modes. It predicts ranges from the full distribution, places hypotheses non-uniformly by equalizing the cumulative probability, and reports the Covering Ratio (the fraction of pixels whose ground truth lies inside the predicted range). UASNet still trains the range and the distribution only through downstream disparity regression; coverage is measured, never optimized. Our proposal keeps UASNet's quantile-style placement, which supplies the method. What changes is what the distribution is trained to do.

On the loss side, there is extensive evidence that smooth surrogates diverge from step-like metrics. Generalized IoU shows that coordinate losses track overlap poorly. Heatmap Distribution Matching proves that a lower heatmap MSE can move the peak further from the target. Direct Loss Minimization and Dembczynski et al. show that standard surrogates can be inconsistent with, or highly regretful under, the true metric. Two stereo papers fix the final-stage version of this problem. Wasserstein Distances for Stereo supervises the whole disparity distribution with an earth-mover loss and predicts from the mode. ADL builds multi-modal targets from neighborhood clustering and uses a dominant-mode estimator. Both operate on a static uniform cost volume and address the final answer, not the handoff between stages. Our target metric differs from theirs. They align the loss with final accuracy; we align it with intermediate coverage, which is the quantity that determines whether final accuracy is reachable at all. Generic metric optimizers such as MetricOpt and UniLoss could in principle target coverage. However, MetricOpt tunes low-dimensional adapters with evolutionary gradient estimates at a reported 22–34% slowdown, and UniLoss needs hand-refactored metrics and anchors near the ground truth. Neither fits a per-pixel, per-stage set objective.

The Coverage Loss

Let $F$ be the coarse stage's predicted cumulative distribution over depth or disparity at a pixel. We use a piecewise-linear cumulative distribution over the bins, so that its inverse is differentiable. The next stage places $K$ hypotheses at fixed quantile levels, $h_k = F^{-1}(q_k)$ for $k = 1, \dots, K$. Each hypothesis can be refined by at most $\delta$, the half-width of the next stage's local search, taken from the architecture rather than tuned. The pixel is recoverable if $\min_k |h_k - d^\ast| \le \delta$, where $d^\ast$ is ground truth. We relax this indicator with a soft minimum at temperature $\tau$:

$$ L_{\text{cov}} = \Big( -\tau \log \sum_{k=1}^{K} \exp\!\big(-|h_k - d^\ast|/\tau\big) \;-\; \delta \Big)_+ . $$

The coarse stage is trained with $L_{\text{cov}}$ plus a small weight on its usual loss. The final stage keeps its standard loss.

The loss has three useful properties. Only the nearest hypothesis is penalized, so mass on a second plausible surface costs nothing as long as one hypothesis lands near the truth; the loss rewards hedging rather than sharpening. Because the hypotheses are quantiles, mass spread into empty space consumes hypotheses that then fail to reach the truth, so diffuse spreading is penalized implicitly through the fixed budget $K$. Finally, $\delta$ ties the objective to what the fine stage can actually repair.

Research Program

The work has three parts. The first is diagnostic. On cascade stereo and on IEBins, we measure where intermediate coverage fails. We test whether the failures concentrate on pixels whose ground-truth neighborhood is multi-modal, using ADL-style clustering as the labeling tool, and whether they grow on Middlebury and ETH3D relative to SceneFlow. If failures are spread uniformly rather than concentrated at boundaries, the multi-modal motivation weakens, and the remaining question is coverage under shift.

The second part is the loss itself. We train it on a cascade stereo network with quantile hypothesis placement, implemented following UASNet's description, and on IEBins using the official code. For IEBins, the single elastic target bin is replaced by $K$ quantile-placed sub-bin centers. IEBins' own authors flag regression-only supervision as a key limitation, so this is a direct test of their conjecture in its intermediate-stage form.

The third part measures the efficiency frontier: coverage and final error as functions of $K$. Better handoffs should allow fewer hypotheses for equal accuracy.

Baselines

The naive reference is a cascade with a fixed-margin window around the coarse estimate, and alongside it a CFNet-style variance-scaled window. For this family, CFNet reports 1.88% D1-all on KITTI 2015. Both are run in our codebase with identical backbones, data, schedules and hypothesis counts. They establish how much coverage failure exists before any learned range.

The strongest existing method is UASNet. It reports 1.66% D1-all on KITTI 2015, 0.53 px end-point error on SceneFlow, and a Covering Ratio "over 99% on KITTI". Since no UASNet code appears in our file, we will reimplement it and first check that we approach these figures. Our loss is then added to the identical architecture, so that the only difference is the coarse-stage training objective. The same seeds, schedule and hyperparameter search budget will be used for both. Beating UASNet means lower end-point error or D1-all on boundary pixels and on zero-shot Middlebury/ETH3D at the same $K$, or matching its accuracy with substantially fewer hypotheses. In-domain KITTI coverage is already near ceiling, so we do not expect gains there and will not claim them.

The final-stage multi-modal baseline applies ADL's multi-modal cross-entropy, and separately the Wasserstein loss (code via W-Stereo-Disp), to the coarse stage of the same cascade. This tests whether a multi-modal distribution target already produces good handoffs without any explicit coverage term. On SceneFlow, ADL reports a 19.59% end-point-error reduction over PSMNet.

Two ablations isolate the key ingredient. The first replaces the set-coverage loss with a single-interval coverage loss: a sigmoid-relaxed indicator that ground truth lies in one predicted interval, plus a width penalty. This separates the multi-hypothesis property from mere direct coverage supervision. The second sets $\delta$ to zero, which turns the objective into a nearest-hypothesis regression and removes its tie to downstream reach.

For monocular depth, the baseline is IEBins as released, which reports 0.050 Abs Rel on KITTI and 85 ms latency. No reported NYU-Depth-v2 number was found on Lacuna for IEBins, so we will report our own reproduction.

Why not just…?

Why not just widen the fixed margin? Wide margins do raise coverage, at a cost in hypotheses and in false matches in textureless regions. UASNet and CFNet both argue that fixed windows fail to transfer across disparity ranges, notably to wide-baseline Middlebury. We will not rely on that argument alone. The efficiency-frontier experiment plots coverage against hypothesis count for fixed margins, so if a margin matches our coverage at equal $K$, the proposal fails on its own terms.

Why not just regress interval bounds with a quantile (pinball) loss and calibrate them with conformal prediction? This is the most serious alternative. It optimizes coverage directly, is simple, and comes with guarantees on average coverage. Its limitation is structural. It produces one interval per pixel, and at a two-surface boundary pixel the interval must either include the empty gap, wasting hypotheses there, or exclude one surface. We will run it as a baseline with the same $K$, spreading hypotheses uniformly or by quantiles within the calibrated interval. If it matches our method on boundary pixels as well as overall, the multi-hypothesis claim is wrong and the contribution reduces to "supervise coverage directly", a weaker result we would report as such.

Why not just train the coarse stage with a multi-modal target like ADL or Wasserstein, since that already keeps mass on both surfaces? It may partly work. These losses fit the target distribution everywhere, including pixels where the truth is unambiguous, and are indifferent to the downstream budget. The coverage loss, by contrast, spends capacity only where a hypothesis would otherwise miss. The multi-modal coarse-stage baseline, alone and combined with our loss, will show whether the effects are redundant or additive.

Why not just use a generic black-box metric optimizer on Covering Ratio? MetricOpt reports a 22–34% training slowdown and acts on low-dimensional adapters, while the coverage decision is made densely per pixel per stage. We will not run it at full scale, and we state this as a scope limitation rather than an experimental result.

Proposed Experiments

Training uses SceneFlow (Mayer et al.) for stereo. Evaluation covers SceneFlow, KITTI 2012/2015, and zero-shot Middlebury and ETH3D. For monocular depth we use KITTI and NYU-Depth-v2 with zero-shot SUN RGB-D. Metrics are intermediate coverage at reach $\delta$, end-point error, D1-all and Abs Rel. Each is reported overall, on boundary pixels identified by multi-modal neighborhoods, and as a curve against hypothesis count $K$.

Three checks decide the outcome. The first is whether coverage failures at boundaries fall under our loss relative to the pinball-plus-conformal baseline at equal $K$; this is the test of the multi-hypothesis claim. The second is whether improved coverage translates into lower final error, rather than coverage rising while the fine stage fails to exploit the extra hypotheses. The third is whether the zero-shot gap to UASNet widens relative to in-domain results, which would support the claim that sharpening-trained coarse stages fail mainly under shift.

A negative result on the first check would still be informative, because it would show that direct single-interval coverage supervision is sufficient. That would be the first such evidence, since no cited work trains intermediate windows for coverage at all.