Paying for Overconfidence in Compute: Calibrated Gates for Iterative Depth Refinement

Many 3D vision systems now fix a quick first guess by running an expensive iterative refiner over it: a diffusion model (a generator that learns by gradually denoising random noise into samples) or a multi-pass network that polishes the result again and again. These refiners help most where a single image is ambiguous, for example in the distance along each camera ray, which a single photograph cannot pin down. They also cost hundreds of milliseconds to seconds per frame, because they spend the same effort on every pixel. The natural fix is to refine only where the first guess is uncertain. That fix fails quietly when the uncertainty estimate is overconfident, because the regions that most need refinement get skipped. We propose to learn a cheap gate that decides, tile by tile, whether refinement is worth running. The gate's threshold is set by a distribution-free calibration procedure that bounds how often a skipped tile would have changed meaningfully under refinement. The calibration target is the refiner's own output, not ground truth, so the gate can be recalibrated on unlabeled frames from a new environment.

This setup lets us measure something the literature has only described: what overconfidence costs once the gate is calibrated. Under a calibrated gate, an overconfident uncertainty signal no longer admits errors. Instead it forces the calibration to refine almost everything, so overconfidence shows up as wasted latency rather than lost accuracy. We expect this cost to concentrate along the depth axis, where single-image ambiguity lives and where both regression training and per-axis variance models shrink uncertainty the most.

Core Claim

Fix a refiner $R$ that maps a first-guess depth map $y_0$ to a refined map $y_T$ after $T$ passes. Split the image into tiles and let a gate score $s_i$ rank tile $i$ by how much refinement is expected to change it. For a threshold $\lambda$, skip every tile with $s_i < \lambda$. Define the per-image loss as the fraction of skipped tiles whose depth would have moved by more than a tolerance $\delta$:

$$L(\lambda) = \frac{1}{|S_\lambda|}\sum_{i \in S_\lambda} \mathbf{1}\big[\lVert y_{0,i} - y_{T,i}\rVert > \delta\big],$$

where $S_\lambda$ is the set of skipped tiles. Conformal risk control picks the smallest $\lambda$ whose average loss on $n$ calibration images, inflated by the standard finite-sample correction, stays below a user budget $\alpha$. This guarantees that expected loss on a new exchangeable image is at most $\alpha$.

We make two claims. First, the guarantee holds at the image level, even though errors are strongly correlated across neighboring tiles, because exchangeability is required only across images. Second, at fixed $\alpha$ and $\delta$, the share of tiles the gate can skip, and therefore the speedup, depends almost entirely on how well the score ranks tiles. Scores taken from variance heads trained with distance losses rank poorly along the ray direction, and they will deliver the smallest speedups.

Why This Is Not Already Done

Several papers use uncertainty to steer refinement, but none decides where or whether to spend refinement compute with a calibrated stopping rule. Perfecting Depth runs ten diffusion samples on every pixel and uses their variance as a mask for a second-stage network. It names that overhead as its main limitation and reports failures in infinite-depth outdoor regions. DifFlow3D refines all scene-flow points with diffusion at 228.3 ms per frame, using a hand-tuned uncertainty threshold. Uncertainty-Aware Diffusion-Guided Refinement of 3D Scenes uses semantic entropy to down-weight hallucinated regions in single-image reconstruction, but every frame still passes through a video diffusion model and a multimodal language model. Pose-Oriented Transformer with Uncertainty-Guided Refinement uses per-joint variance inside a fixed refinement stage, with no calibration and no compute savings. These four papers motivate the proposal: each shows that uncertainty locates the ambiguous regions, and each still pays full refinement cost.

The closest mechanism is Leveraging Neural Radiance Fields for Uncertainty-Aware Visual Localization. It prunes low-confidence 2D–3D matches before a robust pose solver and cuts solver time from 30 ms to 11 ms. That paper prunes inputs to a classical solver using uncalibrated evidential scores, and it assumes a known scene. We gate a neural refiner, give an explicit risk guarantee, and target scenes the system has never seen.

On the calibration side, Object Pose Estimation with Statistical Guarantees wraps keypoint detectors in conformal prediction sets and propagates them to 6D pose, at a cost of several seconds per frame. Variational Inference with Coverage Guarantees in Simulation-Based Inference conformalizes under-dispersed variational posteriors and observes that heavy under-dispersion forces very large prediction sets. We reuse their conformal machinery. The second observation is the efficiency loss we predict will reappear here as lost speedup.

Why Underestimation Becomes a Compute Problem

Two results constrain which gate scores we expect to work. Optimizing MPJPE promotes miscalibration in multi-hypothesis human pose lifting proves that distance-based training losses drive predicted variance toward zero in single-image 3D lifting. On Human3.6M it reports an expected calibration error of 0.42 under minMPJPE training versus 0.07 under negative log-likelihood. Inadequacy of interval estimates corresponding to variational Bayesian approximations and Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes show that treating correlated variables as independent shrinks variance along every axis. Together these results predict that the depth-direction uncertainty of a single-image depth model is understated in two ways: by the loss and by the independence assumption.

Projection makes the problem worse. A pixel's 3D error is elongated along its viewing ray, and the analytic propagation in GUPNet++ shows how small height errors become meter-scale depth errors. Yet GUPNet++ treats its 2D and 3D height errors as independent. MAC-VO shows, in stereo odometry, that a full 3D covariance with cross-axis terms beats a per-axis one by about 10% in rotation error on EuRoC. We carry that lesson into the single-image setting and use it as the basis of one candidate gate score: a lightweight head that predicts a full camera-frame covariance, read off along each ray.

Research Program

The primary testbed is single-image metric depth refinement. A fast feed-forward depth network produces $y_0$. A diffusion-based refiner in the style of Perfecting Depth produces $y_T$. We make the refiner work on tiles by conditioning each masked tile on its unmasked surroundings and blending at tile borders, and we measure how much these seams themselves cost in error. Sky and other infinite-depth regions are handled in inverse depth, so tolerances stay meaningful outdoors.

We compare five families of gate score under the same calibration procedure. The first is a variance head trained with an L1 loss, the kind Pierzchlewicz et al. predict will collapse. The second is the same head trained with negative log-likelihood. The third is the full-covariance head projected onto the ray. The fourth is the magnitude of the refiner's first update, which costs one pass. The fifth is disagreement between two cheap stochastic samples. Each score is calibrated on held-out images of the training domain, and separately on unlabeled images of the test domain. The second option is possible because the loss compares $y_0$ with $y_T$ and needs no ground truth.

We will also study a second decision axis: the number of passes per refined tile. Here a nested calibration bounds the fraction of tiles stopped early whose answer would still have moved by more than $\delta$.

A secondary testbed checks that the mechanism is not specific to depth. We apply per-point gating to DifFlow3D-style scene-flow refinement on KITTI 2015 and FlyingThings3D. A downstream check feeds gated depth into a fixed RGB-D tracking backend on TUM-RGBD and Replica. There, filtering uses the same calibrated scores, and we report trajectory error to test whether skipped tiles damage pose estimation.

Baselines

The naive reference point is the first-guess depth network alone, with no refinement. It sets the latency floor and the accuracy ceiling the gate must close. The opposite reference is full refinement: every tile, the full number of passes. Our closest version of this is a reimplementation of the Perfecting Depth pipeline at its $N=10$ sampling setting, the strongest existing method on this question because it is precisely the uncertainty-aware refiner whose cost we aim to cut. Its reported results are a 21.2% RMSE improvement over raw sensor depth on DIODE and a 289% improvement of its second-stage network at 20% noise on NYUv2. No absolute RMSE or per-frame latency for it was found on Lacuna, so we will measure both ourselves on identical hardware.

For scene flow, the strongest reported baseline is DifFlow3D, at 0.0078 m end-point error on KITTI 2015 and 228.3 ms per frame on an RTX 3090. All methods share the same first-guess network, the same refiner weights, the same calibration images, and the same GPU. Gate heads get an equal, small tuning budget.

Beyond those references, we run three cheaper strategies for cutting refinement cost. A uniform-reduction baseline runs fewer passes everywhere. A random-tile baseline refines a matched fraction of tiles chosen at random. A tuned-threshold baseline sets a raw variance threshold by grid search on validation data, without the finite-sample correction.

Two ablations isolate the key ingredient. In the first, the calibration procedure is held fixed while the score family changes; this isolates ranking quality. In the second, the full-covariance score is held fixed while conformal calibration is swapped for the tuned threshold; this isolates the guarantee.

Beating full refinement means two things. RMSE and threshold accuracy must stay within the tolerance implied by $\delta$ and $\alpha$. Wall-clock latency must be at least twice as fast. The realized risk must stay at or below $\alpha$ both in-domain and after unlabeled recalibration on a shifted domain.

Why not just…?

Why not just tune a threshold on a validation set? Conformal risk control is, mechanically, a validation-set threshold with a correction term, so in-domain the two should behave similarly, and we expect the tuned threshold to be competitive there. The difference shows under distribution shift and small calibration sets. The tuned threshold carries no statement about how often it fails, while ours carries a finite-sample bound, and the ground-truth-free loss lets us recalibrate on unlabeled target frames. The decisive test trains on NYUv2 and deploys on DIODE and KITTI, comparing realized risk to $\alpha$ for both methods. If the tuned threshold also stays within budget across repeated random calibration splits, the guarantee is a modest addition, and the paper's weight shifts to the score-ranking findings.

Why not just stop when the refiner's updates get small? This is the most serious threat, and it is included as the fourth score family rather than argued away. It needs one refinement pass per tile, so its best possible saving is smaller than a gate that runs before refinement. Diffusion updates are also stochastic and need not shrink steadily, which can make a single small update misleading. The score ablation settles this directly. If first-update magnitude matches the full-covariance score in skip rate at equal risk, we will report that the cheapest good gate is the refiner itself and drop the covariance head.

Why not just distill the refiner into one or two steps? Single-pass distillation, as in DCFold for protein structure, cuts cost per tile, whereas gating cuts the number of tiles, so the two should stack. We will test gating on top of a reduced-step refiner. If the distilled refiner alone already matches full-refinement accuracy at gated latency, spatial gating adds little in that regime, and we will say so.

Why not just use ensembles or Monte Carlo dropout for better uncertainty? These methods may improve ranking. But each sample is another forward pass, and that cost is exactly what Perfecting Depth identifies as prohibitive. We include a two-sample disagreement score as the cheapest member of this family and report its latency honestly, including the samples.

Proposed Experiments

The central experiment plots skip rate and latency against realized risk for each score family, on NYUv2, DIODE, ScanNet and KITTI, at several values of $\alpha$ and $\delta$. Our prediction is specific and falsifiable. The L1-trained variance head will need the lowest threshold to meet the budget and will therefore skip the fewest tiles. The negative-log-likelihood head will do better. The ray-projected full-covariance score will do best on outdoor scenes with long depth ranges. If the L1 head's speedup is not clearly worse, the claim that underestimation shows up as lost compute is wrong.

A second check measures realized risk after unlabeled recalibration in a new domain, against risk when the in-domain threshold is reused unchanged. This tests whether the ground-truth-free loss actually protects against the out-of-distribution threshold failures noted throughout the filtering literature.

A third check repeats the gating on KITTI 2015 scene flow, reporting end-point error against DifFlow3D's 0.0078 m and per-frame latency against its 228.3 ms. A fourth feeds gated depth into a fixed tracking backend on TUM-RGBD and Replica and reports trajectory error, to confirm that a bound stated in depth space does not hide pose-level damage.

The proposal fails if no score skips more than a small fraction of tiles at a useful $\alpha$, or if seam artifacts from tile-wise refinement cost more accuracy than the skipped refinement would have recovered. Either outcome would itself be informative about how spatially concentrated single-image depth ambiguity really is.