Neural radiance fields render a pixel by evaluating a neural network at many points along the camera ray and summing the results, and this repeated evaluation is the main reason they are slow. A popular shortcut asks an off-the-shelf depth estimator where the surface probably is, then samples only a narrow window around that guess. The shortcut fails badly when the depth guess is wrong. On shiny, reflective, or textureless surfaces the window can miss the true surface entirely, and no later optimization can recover geometry that was never sampled. We propose to replace the single depth guess with a calibrated depth interval for each ray: a range that is statistically guaranteed to contain the true surface with a chosen probability, say 95%. A fixed sample budget is then spread across rays in proportion to how wide their intervals are. Confident rays receive a handful of samples. Uncertain rays receive wider and denser coverage.
The calibration is the hard part. A depth model's own sense of its error is unreliable exactly where it matters, on unusual materials and under domain shift. We therefore train the interval predictor on procedurally generated synthetic scenes whose materials are deliberately randomized to provoke depth failures, because these scenes come with perfect ground-truth depth. We then correct the residual synthetic-to-real gap with a small set of real scans by conformal calibration, a simple statistical procedure that turns any interval predictor into one with guaranteed coverage on data like the calibration set. The claim is concrete. At equal compute, calibrated intervals should sharply reduce the fraction of rays whose surface falls outside the sampled range. That reduction should translate into better rendering on hard materials, at no cost on easy scenes.
Core Claim and Positioning
This is a two-stage pipeline: a depth prior feeds a sampler. It inherits the classic ceiling of such pipelines, where downstream recall cannot exceed upstream recall. That ceiling was shown for 2D-to-3D object detection by Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses, and in rendering it takes a precise form: rays whose surface lies outside the sampled interval are unrecoverable. Our proposal treats the probability of that miss as the quantity to control.
Two existing methods motivate this framing because they hit the ceiling directly. DONeRF trains a classifier that picks a depth bin and places 4 samples around it, cutting compute by 24× to 98×. It treats the bin as a hard decision and assumes one opaque surface per ray. DARF centers a narrow sampling window on a Depth Anything v2 prediction and refines it, halving samples. It reports failures on specular scenes where the prior is wrong.
TermiNeRF and IDESplat go further and predict distributions over depth: ray-termination distributions and iteratively refined depth probabilities, respectively. Neither checks whether those distributions are calibrated, and neither uses them to set a per-ray sample budget. The calibration gap is the one documented by On the Calibration of Human Pose Estimation. That paper shows raw network confidences are miscalibrated against true geometric error, and that recalibrating them improves downstream 3D fitting. It is the motivating evidence for our design, but its fix (a small network on frozen features) carries no coverage guarantee and was never tested under domain shift.
PRISM-SLAM supplies the methodological stance we reuse. It feeds foundation-model depth into a factor graph as a probabilistic prior rather than a hard constraint, and down-weights unreliable pixels softly. We carry that stance from pose optimization into sample allocation. We also use the predicted spread as an inverse-variance weight on the depth-prior regularizer during radiance-field training. This is the standard uncertainty-weighted least-squares form, so that an unreliable prior pulls less on the geometry.
Research Program
The interval predictor is a lightweight head attached to a DARF-style generalizable radiance field. Its inputs are the monocular depth prediction, image features, and a multi-view photometric-consistency signal computed by projecting the predicted depth into the source views. That last input matters: on mirror-like surfaces the monocular prior can be confidently wrong, but the source views will disagree with it. For each ray $r$ the head outputs lower and upper depth quantiles. It may optionally output a second interval, for rays crossing transparent or reflective layers where one surface is not enough. Training uses a quantile (pinball) loss against ground-truth depth.
The training scenes are synthetic. Following the domain-randomization logic of Domain Randomization for Scene-Specific Car Detection and Pose Estimation and the diversity-over-realism finding of Diversity is Definitely Needed, we randomize the material properties that break depth estimators: specular strength, transmission, textureless albedo, and lighting. The goal is not photorealism. The goal is for the head to see many examples of the depth prior failing, each with an exact answer, which real datasets rarely provide on shiny surfaces.
Because synthetic training alone will leave some miscalibration on real images, we apply split conformal calibration on $m$ held-out real scans with ground-truth depth. Let $s_r$ be the amount by which ray $r$'s true depth falls outside its predicted interval, which is zero if inside. We widen every interval by the $\lceil (m'+1)(1-\alpha) \rceil$-th smallest score among the $m'$ calibration rays, where $1-\alpha$ is the target coverage. Conformal prediction then guarantees that coverage holds marginally on exchangeable data. The guarantee is only marginal and per ray, and real rays within an image are correlated, so we empirically test how the guarantee degrades across scenes and across datasets.
Sample allocation then solves a budget problem. Given a total of $B$ network evaluations per image and calibrated interval widths $w_r$, we assign
$$n_r = \max\left(n_{\min},\ \left\lfloor B\, \frac{w_r}{\sum_{r'} w_{r'}} \right\rfloor\right),$$
where $n_{\min}$ is a floor of about 2 to 4 samples. This keeps the spacing between samples roughly constant across rays, which is the natural target when quadrature error grows with spacing. Rays with two intervals split their samples between them.
Baselines
The naive reference is full hierarchical stratified sampling, as in NeRF and MVSNeRF, run at their native budgets. This gives a quality ceiling at high cost. The same samplers will also be run at our reduced budget to show what naive cutting loses.
The strongest existing competitor is DARF, because it attacks exactly this problem with exactly this depth prior. It reports 33.28 dB PSNR on DTU with at least 50% fewer samples (DARF). We will retrain DARF and our method from the same backbone, on the same training scenes, with the same Depth Anything v2 prior and equal tuning budgets. Compute is matched by counting network evaluations per image and by wall-clock time. Beating DARF means one of two things. Either we match its DTU PSNR within noise at equal or lower budget while winning clearly on the specular and textureless subsets, or we trace a PSNR-versus-samples curve that lies above DARF's.
The skeptic's baseline is DARF with a fixed safety margin: its window widened by a constant, plus a few uniform samples spread over the whole ray, with margin and count tuned on the same calibration scans we use. A second uncertainty baseline replaces our learned head with the disagreement among Depth Anything v2 predictions under test-time augmentations, calibrated conformally in the same way. DONeRF's deterministic depth-bin oracle at 4 samples is included on its own synthetic benchmark.
Three ablations isolate our ingredients. The first removes the conformal step. The second trains the head on real data only, with no randomized synthetic scenes. The third spreads samples uniformly across rays inside the calibrated intervals instead of in proportion to width. For comparison with point-based methods, IDESplat is reported where its benchmarks overlap. It reports 27.80 dB on RealEstate10K (IDESplat).
Why not just…?
Why not just widen DARF's window by a fixed margin and add a few uniform samples? This is the real threat. On DTU, where the depth prior is mostly right, a tuned fixed margin may do as well as calibrated intervals. Our hypothesis is that a fixed margin is either too wide everywhere, wasting compute on easy rays, or too narrow on the rare rays where the prior is far off. The fixed-margin baseline is one point on the compute-quality trade-off, so the decisive test is whether our curve beats it on the specular and textureless subsets. If it does not, the honest conclusion is that per-ray adaptivity buys nothing over a tuned global margin, and we would report that.
Why not just use the depth model's own disagreement under augmentation as the uncertainty? Augmentation disagreement measures sensitivity to input perturbations. It does not measure the systematic errors of a model that is confidently wrong on a mirror, and confident wrongness is precisely the overconfidence pattern On the Calibration of Human Pose Estimation documents for raw confidences. The multi-view consistency input is our answer to this case. The ensemble baseline, conformally calibrated on equal terms, tests whether that input is needed.
Why not just use the coarse-to-fine sampling built into NeRF, with no external prior? The coarse pass is itself a dense sweep along every ray, so it pays much of the cost we are trying to avoid. DONeRF and DARF exist because that cost is large. The full-budget hierarchical baseline shows the quality ceiling. The reduced-budget version shows that hierarchy without a reliable prior degrades quickly.
Why not just switch to 3D Gaussian Splatting, which avoids ray marching altogether? In feed-forward splatting the Gaussian centers come from predicted depth, so the same failure appears in a different place: IDESplat builds its method around this depth-to-Gaussian error cascade. Calibrated intervals could equally decide how many Gaussians to place along an uncertain ray. We treat that extension as follow-up work rather than a refutation.
Proposed Experiments and Decisive Checks
Training uses the randomized synthetic scenes plus the standard generalizable-NeRF training splits. Conformal calibration uses a few DTU training scans with ground-truth depth.
Evaluation covers four datasets. DTU tests in-distribution performance, using PSNR, SSIM, LPIPS, depth error, and samples per ray. LLFF tests real forward-facing scenes. Shiny Blender is the specular stress test DARF identifies; no reported result was found on Lacuna for DARF on it. Spaces tests wide-baseline outdoor scenes, where SC-NeRF reports 25.99 dB PSNR against 19.37 for MVSNeRF (SC-NeRF). The DONeRF synthetic scenes serve as the multi-surface test.
Beyond image quality, we report the miss rate: the fraction of rays whose ground-truth depth lies outside every sampled interval. We also report the gap between target coverage and achieved coverage on held-out scans.
Three outcomes would decide the proposal. First, achieved coverage on held-out DTU scans and on Shiny Blender should stay close to the target. If it collapses on the specular data, synthetic randomization has failed to transfer and the method's premise is wrong. Second, at matched compute, miss rate and PSNR on the specular and textureless subsets should beat both DARF and the tuned fixed-margin baseline, while DTU-wide PSNR stays within noise of DARF. Third, per-ray interval width should correlate with per-ray rendering error. If it does not, the budget is being spent on the wrong rays, whatever the averages say.
References
- Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses
- DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
- DARF: Depth-Aware Generalizable Neural Radiance Field
- TermiNeRF: Ray Termination Prediction for Efficient Neural Rendering
- IDESplat: Iterative Depth Probability Estimation for Generalizable 3D Gaussian Splatting
- On the Calibration of Human Pose Estimation
- PRISM-SLAM: Probabilistic Ray-Grounded Inference for Scale-aware Metric SLAM
- Domain Randomization for Scene-Specific Car Detection and Pose Estimation
- Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion
- NeRF: representing scenes as neural radiance fields for view synthesis
- MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
- SC-NeRF: Self-Correcting Neural Radiance Field with Sparse Views