Many of the best single-image depth estimators no longer regress a distance per pixel directly. They classify it: the distance range is cut into intervals ("bins"), the network predicts a probability for each bin, and the answer is refined by zooming into the most likely bin over several rounds. Every pixel receives the same number of rounds. That includes a pixel on a blank wall, which is correct after the first round, and a pixel on the edge of a thin pole, which is still uncertain after the last. This proposal makes the number of refinement rounds depend on the region of the image. Refinement stops, tile by tile, once the remaining search interval shows that further rounds cannot move the answer by a meaningful amount. The rounds saved on settled tiles are then spent on unsettled ones. The stopping threshold is calibrated so that, with a stated probability, halting changes almost no pixel's depth relative to running every round.
The opportunity is specific to classification-style depth. In language models, early-stopping signals such as answer agreement or probe scores are only indirect evidence that more computation will not help. In a coarse-to-fine bin search, the current interval width is a direct, physical bound on how far later rounds can move a prediction, apart from the explicit widening the method performs when uncertain. The depth side supplies a halting signal that is meaningful in its own right. The compute-allocation side supplies the calibration machinery and the lesson that coarse, batched allocation beats fine-grained, irregular allocation on real hardware.
Core Claim
IEBins is the method this proposal builds on. It searches 16 bins per stage. It subdivides the chosen bin at each stage and widens that bin "elastically" when the predicted distribution has high variance, which prevents early mistakes from locking in. A recurrent unit carries state between stages. On KITTI it reports an absolute relative error of 0.050 at 85 ms, against 216 ms for BinsFormer at a similar roughly 273M parameters. Its iteration count is fixed across images and pixels.
We claim that a per-tile statistic $s = w \cdot u$ is a reliable, cheap halting rule. Here $w$ is the width of the tile's current search interval after elastic widening, and $u$ is one minus the probability mass on the selected bin, averaged over the tile's pixels. Thresholding $s$ at a level chosen by Learn-then-Test calibration should cut total refinement work substantially on NYU Depth v2 and KITTI with no measurable loss in global metrics. Moving the saved rounds to high-$s$ tiles should reduce error in a band around depth discontinuities.
The contrast with the closest work is explicit. Bi3D trades accuracy for latency by choosing how many binary depth planes to test. It lists content-driven adaptive plane sampling as future work, and its allocation is set manually and applied uniformly across the image. AdaBins and BinsFormer adapt where the bins are placed, but not how much computation each region receives. None of the three decides, per region and during inference, when refinement is finished.
Research Program
The mechanism has three parts.
Tile-level halting. The image is divided into feature tiles, for example 8×8 cells at the recurrent unit's resolution. We halt tiles rather than individual pixels because of the hardware concern raised in the file: irregular per-pixel branching destroys GPU efficiency. The two works that faced this problem solved it by grouping. Escape Sky-high Cost batched its stopping checks into windows. Learning How Hard to Think found coarse difficulty pools more stable than fine online allocation. At each stage, active tiles are gathered into a dense tensor and run through the recurrent update. Halted tiles keep a frozen hidden state and their current depth, and the results are scattered back.
Training for variable-length refinement. The recurrent unit must behave sensibly when a tile stops at round two or runs past the original count. We therefore fine-tune from the public IEBins checkpoint (ShuweiShao/IEBins). Iteration counts are randomized per tile during training, and the loss is applied at every stage.
Calibrated threshold. We borrow the calibration from Thought calibration, which uses Learn-then-Test to choose a stopping threshold on reasoning-token probes with a finite-sample risk bound. Our risk is defined as the fraction of valid pixels whose depth under halting differs from the full-budget depth by more than a relative tolerance $\epsilon$. We calibrate the threshold $\lambda$ on $s$ on held-out NYU and KITTI splits so that this risk stays below a user-set level $\alpha$ with probability $1-\delta$. The guarantee is relative to the full-budget model, not to ground truth. That choice is deliberate: it certifies "halting costs nothing," which is the claim we want.
We also analyze which quantity actually predicts the benefit of further refinement. The candidates are the interval width alone, the entropy alone, their product, and a learned two-layer probe on the recurrent state. The learned probe follows the marginal-reward predictors of Learning How Hard to Think and the hidden-state difficulty probes of The LLM Already Knows. If a learned probe clearly beats the physically motivated statistic, we adopt the probe and report that the interval bound is not enough.
Baselines
The naive reference point is IEBins with fewer iterations applied uniformly. We run it at every total budget our method reaches, measured in average tile-rounds per image. This gives the uniform-allocation accuracy-versus-compute curve. Beating it means strictly lower Abs Rel and boundary-band error at matched budget and matched measured latency.
The strongest existing method is full-budget IEBins itself, which reports KITTI Abs Rel of 0.050 at 85 ms. It will be re-run from the official code on the same hardware so that latencies are comparable. Beating it means matching its metrics within noise at a clearly lower measured latency. The reallocation variant should, in addition, lower error in the boundary band at equal latency.
For historical context, DORN reports δ<1.25 of 0.828 on NYU Depth v2. The file quotes no NYU numbers for AdaBins, BinsFormer, or IEBins, so we will report our own re-runs for those. BinsFormer is not re-run here; we include it only through the 216 ms latency IEBins reports for it.
A heuristic spatial allocation baseline assigns extra rounds to tiles with strong image gradients or large first-round depth gradients, at the same budget. This tests whether "compute at edges" needs anything learned or calibrated at all.
An uncalibrated entropy threshold halts a tile when its bin entropy falls below a hand-set value, in the style of consensus-based stopping in Escape Sky-high Cost. It isolates the contribution of both the width term and the calibration.
The ablations are: halting without reallocation; reallocation without the variable-length training; per-pixel versus tile halting, measuring wall-clock time and not just floating-point operations; and an oracle that halts each tile exactly when further rounds would stop changing its depth. The oracle bounds the achievable savings.
All variants share the IEBins backbone, training data, fine-tuning schedule, and hyperparameter search budget.
Why not just…?
Why not just run fewer iterations everywhere? This is the most serious competitor. If the accuracy lost by uniform truncation is tiny, adaptive halting has little to offer. The matched-budget curve decides the question directly. Our method is justified only if its curve lies clearly below the uniform-truncation curve at moderate budgets, and especially if uniform truncation degrades the boundary band while adaptive halting does not. If the two curves coincide, we will report that the recurrent refinement in IEBins is over-provisioned and not spatially heterogeneous.
Why not just send compute to image edges with a fixed detector? Image edges and depth discontinuities differ. Textured planar floors have strong image gradients but settled depth. Low-contrast occlusions have weak image gradients but unsettled depth. The equal-budget heuristic baseline measures how much this mismatch matters. If the heuristic matches the calibrated statistic, the simpler rule wins and we will say so. It still lacks a risk guarantee, however.
Why not just use a lightweight static network? The plane-residual depth completion model shows that compact static models can be fast, running at 100 FPS with 14M parameters. It solves a different task, with LiDAR input, and it reduces capacity everywhere, including at hard pixels. Our question is whether a large model's capacity can be kept where it is needed. The lightweight-backbone comparison belongs on the latency axis, and we include it only as context.
Why not just threshold entropy, as consensus stopping does? The file itself warns about the "consistently wrong" failure. A tile can be confidently in the wrong bin, which is exactly the case elastic widening exists to handle. Entropy ignores whether the interval has been widened. The product $s$ does not, because widening raises $w$. The uncalibrated-entropy baseline and the oracle comparison show whether this distinction matters in practice.
Proposed Experiments
We will train and evaluate on NYU Depth v2 (dense ground truth, which makes a boundary band computable from ground-truth depth discontinuities) and on the KITTI Eigen split. We report Abs Rel, RMSE, δ<1.25, and error inside a band a few pixels wide around discontinuities. Compute is reported as average tile-rounds and as measured latency on a single fixed GPU.
The first decisive check is the matched-budget comparison against uniform truncation.
The second is calibration validity. On held-out in-distribution data, the empirical risk should stay below $\alpha$. On zero-shot SUN RGB-D after training on NYU, we expect the bound to loosen, since exchangeability breaks. This mirrors the smaller savings Thought calibration reports out of distribution: 20% versus up to 60%. We will measure how far the realized risk drifts and whether recalibrating on a small SUN RGB-D split restores it.
The third is whether tile gathering preserves real speedups. Measured latency must fall roughly in proportion to active tiles; otherwise the floating-point savings are illusory.
The fourth is a negative control. On images that are mostly planar, the method should halt almost everything after one or two rounds with no accuracy change. On cluttered scenes it should approach the full budget.
References
- IEBins: Iterative Elastic Bins for Monocular Depth Estimation
- Bi3D: Stereo Depth Estimation via Binary Classifications
- AdaBins: Depth Estimation using Adaptive Bins
- BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- Thought calibration: Efficient and confident test-time scaling
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Deep Ordinal Regression Network for Monocular Depth Estimation
- Depth Completion Using Plane-Residual Representation