On a white wall, multi-view photometric matching gives a 3D reconstruction method almost no information. Current systems fill the gap by asking a pretrained single-image network for depth and surface normals and pulling the 3D model toward those predictions everywhere. When that network is wrong, for example on mirrors, glossy floors, or the thin edge where a wall meets a cabinet, the reconstruction copies its mistake. We propose to change the supervision. Instead of a single target, each pixel gets an uncertainty set: a cone of normal directions, and an interval of depths, that the true surface is guaranteed to lie in with a chosen probability. The 3D model is penalized only when it leaves that set. The sets are built by conformal prediction, a distribution-free calibration method that turns a predictor's errors on held-out data into prediction regions with finite-sample coverage guarantees. Their width depends on how informative the image is locally, so the sets are tight where the prior is reliable and wide where it is not.
The idea comes from worst-case (robust) optimization in wireless communications. There, an imperfect channel estimate is treated as the center of a bounded uncertainty region, and the design must satisfy its constraint for every channel in that region. The robust counterpart of "the true surface lies in $\mathcal{U}(p)$" is a dead-zone constraint: the rendered geometry must be feasible for the set, and nothing more is demanded. Worst-case optimization also gives us the right way to evaluate. Average error hides the failures that matter, so we judge methods on the worst regions of each scene.
Core Claim
Point-target prior losses are the main channel through which monocular errors reach 3D geometry. Replacing them with calibrated, spatially varying feasibility sets should do two things. On textureless planar regions it should keep the completeness gains of priors, because the sets there are tight. On regions where the prior is wrong it should sharply reduce tail geometric error, because the sets there are wide and multi-view photometry takes over. What is new is the set itself: a cone and interval with a coverage guarantee, calibrated on held-out scenes and conditioned on local photometric informativeness. The hinge-style penalty that enforces the set is deliberately simple. We say so up front, and our ablations isolate the set construction from the penalty.
Relation to Prior Work
Three works supply the robust-optimization method. Robust Cognitive Beamforming With Partial Channel State Information and Worst-Case SINR Constrained Robust Coordinated Beamforming for Multicell Wireless Systems model estimation error as ellipsoids around a nominal estimate. They enforce constraints for every realization in the ellipsoid, using tractable reformulations (the S-procedure and semidefinite relaxation). Conformal Robust Beamforming Via Generative Channel Models replaces hand-chosen ellipsoids with conformal prediction sets around a learned estimator. We reuse that last move directly, with one change: the uncertain quantity is a per-pixel surface normal and depth, not a channel matrix. A normal cone, the set of directions within angle $\theta(p)$ of the predicted normal, is the spherical analogue of their uncertainty regions. Membership in it is a single scalar inequality on a dot product, so no relaxation machinery is needed. The beamforming literature also warns about conservatism. Worst-Case SINR Constrained Robust Coordinated Beamforming for Multicell Wireless Systems reports that robust designs pay extra power under nominal conditions. Our analogue of that cost is that sets which are too wide hand textureless walls back to the shape-radiance ambiguity. This constrains the design: the sets must be conditioned on region type, not uniform.
Two works motivate the proposal. MonoSDF supervises a neural signed distance function with Omnidata depth and normals using L1 and angular losses, and acknowledges that accuracy is capped by the external predictor. DN-Splatter brings the same kind of priors into 3D Gaussian Splatting, a representation built from many small semi-transparent ellipsoids. It uses edge-aware depth losses and flattens the Gaussians along the predicted normals. Both treat the prior as pseudo ground truth. We keep both pipelines and change only what the prior is allowed to demand.
Several works constrain or compete with the proposal. 3D Gaussian Flats snaps Gaussians onto planes found by RANSAC on segmentation masks. StructNeRF imposes superpixel coplanarity. Planar Prior Assisted PatchMatch Multi-View Stereo propagates planar hypotheses from a triangulation of reliable matches. All three replace an unreliable prior with a hard structural assumption. A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos cautions that such assumptions earn completeness by hallucinating flat geometry. Geo-Neus uses sparse structure-from-motion points and patch normalized cross-correlation. We reuse this as the photometric term that must take over inside wide sets. Probabilistic reconstruction work estimates uncertainty, as in ActiveNeuS and Stochastic Poisson Surface Reconstruction with One Solve using Geometric Gaussian Processes. These methods quantify uncertainty in the reconstruction itself. None of them calibrates the input prior against held-out ground truth or gives a coverage statement. The research file found no work applying bounded uncertainty sets to prior-supervised textureless reconstruction.
Research Program
Building the sets. We first build calibrated sets for an off-the-shelf monocular normal and depth estimator. The Omnidata models are the ones used by MonoSDF and DN-Splatter. Calibration scenes are a training split of ScanNet++, which has laser-scanned ground truth. Each image is partitioned into regions using two signals: a local photometric informativeness score (patch gradient energy and multi-view normalized cross-correlation peakedness) and the predicted normal's local variance. Regions are then pooled into a small number of bins. Within each bin we compute a nonconformity score, which is the maximum angular normal error over the region. Using a region-level maximum is the worst-case choice. It also makes the calibration units closer to exchangeable than individual pixels, whose errors are strongly correlated within a scene. Group-conditional split conformal calibration then gives a per-bin angle $\theta_b$. The guarantee is that, for a new region from an exchangeable scene, the region's maximum error is at most $\theta_b$ with probability at least $1-\alpha$. Depth is affine-ambiguous from a single image, so for depth we calibrate the residual after a robust scale-shift fit, giving a per-bin interval half-width $\delta_b$.
Using the sets in reconstruction. With $\hat n(p)$ the predicted normal, $n_\Theta(p)$ the rendered normal of the 3D model with parameters $\Theta$, and $b(p)$ the bin of pixel $p$, the normal term becomes
$$ L_{\text{set}}(\Theta) = \sum_p \max\big(0,\ \cos\theta_{b(p)} - \hat n(p)^\top n_\Theta(p)\big), $$
which is zero whenever the rendered normal lies inside the calibrated cone. The depth term is the analogous dead zone of width $\delta_{b(p)}$ around the aligned prior. The photometric and patch-consistency terms are unchanged. Inside a wide set, they and the representation's own regularizers decide the geometry. We will also test a variant in which the scale-shift alignment is itself made robust: we fit it only on pixels whose sets are tight, so wrong priors cannot corrupt the alignment. The same loss is inserted into DN-Splatter and into MonoSDF to show that the effect does not depend on the representation.
Analysis. We will measure how coverage degrades under scene shift and how conservatism trades off against tail error as $\alpha$ varies. We will also test whether the informativeness-conditioned bins actually separate reliable from unreliable prior regions, by checking whether set widths correlate with true prior error on held-out scenes.
Baselines
The naive reference point is each backbone with no geometric prior: standard 3D Gaussian Splatting, and the VolSDF-style signed distance model without cues. This shows how much priors matter at all. The file reports standard 3DGS at an F-score of 0.5835 on MuSHRoom (DN-Splatter) and a depth RMSE of 0.44 on ScanNet++ (3D Gaussian Flats).
The strongest direct competitor is DN-Splatter itself. It uses exactly the prior we use, in exactly the representation we modify, so any gain is attributable to how the prior is used. It reports an F-score of 0.9243 on MuSHRoom (DN-Splatter). We will rerun it from its released configuration on identical images, poses, and Omnidata outputs, with the same iteration count. Its loss weights and our $\alpha$ get the same tuning budget on the same validation scenes. For the implicit-surface track, MonoSDF is the matched competitor, with a reported ScanNet Chamfer distance of 0.042 for its MLP variant (MonoSDF). 3D Gaussian Flats is the strongest structural-prior competitor on ScanNet++, with a reported depth RMSE of 0.27 (3D Gaussian Flats).
Beating these baselines means two things together. First, matching their mean F-score and Chamfer distance within noise. Second, reducing the 95th-percentile region-level geometric error, and the error on regions where the prior is wrong, by a margin larger than the variation across random seeds.
The key ablation replaces calibrated per-bin sets with a single hand-tuned constant margin, an ordinary hinge loss, tuned with the same budget. A second ablation uses calibrated but unconditioned sets (one bin). These two isolate the contribution of calibration and of conditioning on informativeness.
Why not just…?
Why not just keep the L1 and angular losses and tune their weights? Under a point-target loss, a wrong prior keeps pulling the surface toward its own error. Lowering the global weight reduces that pull everywhere, including on walls where the pull was the whole benefit. MonoSDF's own statement that accuracy is capped by the predictor (MonoSDF) points at exactly this. Our comparison against DN-Splatter, run under an equal tuning budget and split by prior-error regions, will show whether a single scalar weight can recover the tail error. If it can, the proposal fails.
Why not just use a robust loss or a constant-margin hinge? This is the most serious threat, because our penalty is a hinge. The claim rests entirely on the margin being calibrated and spatially varying. The constant-margin ablation tests this directly. If a well-tuned constant margin matches calibrated sets on both mean and tail metrics, we will report that the robust-optimization framing adds nothing beyond a dead-zone loss.
Why not just learn per-pixel uncertainty jointly and down-weight high-variance pixels? Learned aleatoric weights are fit to the same scene they supervise. They can therefore learn to discount exactly the textureless pixels where the prior is needed. They also carry no coverage statement. Uncertainty-weighting methods in the file, such as ActiveNeuS, use uncertainty for down-weighting or view selection rather than as calibrated bounds. We will include a learned-variance weighting variant on DN-Splatter under the same budget. We will also test whether its implied intervals cover held-out ground truth at the nominal rate on scenes it was not fit to.
Why not just snap textureless regions to planes? Planar snapping in 3D Gaussian Flats depends on a warm-up reconstruction and on segmentation masks, and it slows training by 1.5×. The ETH3D authors warn that hard planarity trades accuracy for hallucinated completeness (A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos). Calibrated sets impose no structure where the prior is uncertain. The ScanNet++ comparison, restricted to non-planar textureless regions, will settle this.
Proposed Experiments
Calibration uses ScanNet++ training scenes. Testing uses held-out ScanNet++ scenes as an in-distribution check, MuSHRoom and ScanNet as shift tests, and Replica for clean normal evaluation. We report the backbones' standard metrics: F-score and Chamfer distance on meshes, depth RMSE, and normal angular error. We add three worst-case metrics: 95th-percentile region error, error restricted to regions where the prior's true error exceeds 20°, and error restricted to low-informativeness regions.
The first decisive check is empirical coverage. On held-out and shifted scenes, region-level coverage must be close to $1-\alpha$. If coverage collapses on MuSHRoom or ScanNet, the guarantee does not transfer, and the method reduces to a heuristic. The second check is that tail error on prior-wrong regions falls relative to DN-Splatter and MonoSDF, while mean F-score stays within noise of 0.9243 on MuSHRoom. The third check is the conservatism test: error on low-informativeness planar regions must not rise. A rise would mean the sets are too wide, and textureless walls have been returned to the shape-radiance ambiguity. The fourth check is the constant-margin and learned-variance ablations, which determine whether calibration is the ingredient doing the work. Sweeping $\alpha$ traces the trade-off between conservatism and tail error, the reconstruction analogue of the power cost of robust beamforming.
References
- Robust Cognitive Beamforming with Partial Channel State Information
- Worst-Case SINR Constrained Robust Coordinated Beamforming for Multicell Wireless Systems
- Conformal Robust Beamforming Via Generative Channel Models
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
- DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing
- 3D Gaussian Flats: Hybrid 2D/3D Photometric Scene Reconstruction
- StructNeRF: Neural Radiance Fields for Indoor Scenes With Structural Hints
- Planar Prior Assisted PatchMatch Multi-View Stereo
- A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos
- Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction
- ActiveNeuS: Active 3D Reconstruction using Neural Implicit Surface Uncertainty
- Stochastic Poisson Surface Reconstruction with One Solve using Geometric Gaussian Processes