Uncertainty-Shaped Gaussians for Single-Image 3D Reconstruction

A single photograph does not fix how far away anything is. Every pixel could have come from any point along its line of sight. Feed-forward reconstruction models are networks that turn images into a 3D scene in one pass instead of fitting each scene by gradient descent. When they work from one image, they paper over this ambiguity. They place a sharp, confident 3D blob at one guessed depth per pixel, and sometimes add a fixed stack of extra blobs behind it for hidden background. When the guess is wrong, the scene looks right from the input camera and breaks apart as soon as the camera moves. Separately, work on the reliability of 3D networks has shown that these depth guesses are badly overconfident. Nothing in current systems uses that knowledge to decide what the 3D scene should look like.

We propose to make depth uncertainty part of the 3D representation itself. For each pixel, a network predicts a small, calibrated distribution over depth along the camera ray. That distribution decides two things. It sets the shape of each primitive: a Gaussian stretched along the ray in proportion to how unsure the network is. It also sets how many primitives the pixel gets: one where depth is certain, and several only where the distribution has multiple modes or where hidden surfaces are likely. The result is a single-image reconstructor whose primitive budget follows geometric ambiguity rather than image resolution. As a side effect, it can render an honest per-pixel depth uncertainty map for any new viewpoint.

Core Claim

Two claims are tested separately. The first is that, at a matched budget of primitives, spending Gaussians where calibrated depth uncertainty is high beats spending them uniformly. The second is that stretching primitives along the ray by calibrated depth variance produces rendered depth intervals that actually contain the true depth at the stated rate, without losing image quality at moderate camera shifts.

Some terms are needed. A 3D Gaussian here is a semi-transparent ellipsoid with a center, covariance (shape), opacity and color; scenes made of many of these are drawn by fast rasterization, a technique called Gaussian splatting. For pixel $p$ with viewing direction $r_p$, the network outputs up to $K$ candidate depths. Candidate $k$ has a weight $w_{pk}$, a mean depth $d_{pk}$ and a standard deviation $\sigma_{pk}$. A candidate becomes a Gaussian only if $w_{pk}>\epsilon$. Its covariance is

$$\Sigma_{pk} = R_p\,\mathrm{diag}(s_{pk}^2,\ s_{pk}^2,\ \lambda\,\sigma_{pk}^2)\,R_p^\top ,$$

where $R_p$ is a rotation whose third axis is $r_p$, $s_{pk}$ is the learned sideways footprint, and $\lambda$ is a single global scale fitted so the depth spread is calibrated. Training minimizes a rendering loss on target views, plus a negative log-likelihood of pseudo-depth under the depth mixture, plus a budget penalty $\beta\sum_{p,k} w_{pk}$. The budget penalty pushes the network to spend extra candidates only where they pay off.

Relation to Prior Work

The closest single-image system is Flash3D. It lifts one image into layered Gaussians anchored on UniDepth. It gives every pixel the same number of layers and constrains their depth offsets to be non-negative, so an overestimated foreground depth can never be corrected. We reuse its setting and depth anchor. We differ in three ways: layers are adaptive, offsets are signed and probabilistic, and primitive shape comes from uncertainty.

The closest mechanism is pixelSplat. For image pairs, it predicts a discrete depth distribution per pixel, samples one depth, and sets the Gaussian's opacity to that probability. That is a trick for getting useful training gradients. It still spawns exactly one primitive per pixel, is never calibrated, and needs two views. We use the distribution to control how many primitives exist and how they are shaped, and we evaluate whether it is calibrated.

The motivation comes from two uncertainty papers. Trust3R predicts evidential 3D covariances for pointmaps from networks such as DUSt3R, but its output is unorganized points with no rendering and a single-mode distribution per pixel. A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction finds a median 2.4x overconfidence across backbones. It shows a two-parameter power-law fix brings this to 1.11x, but applies the fix only after the fact to pointmaps. Our global $\lambda$ is the rendering-space counterpart of that fix. The audit's finding that confidence ranks pixels well but has the wrong scale is exactly why one fitted scalar can plausibly work.

Two papers constrain the design. Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting shows that depth discontinuities at object silhouettes tear unprojected geometry. Silhouettes are exactly where a mixture over depth should put two modes, one for foreground and one for background, instead of one smeared depth. We also borrow its pointmap-alignment idea: pseudo-depth targets come from VGGT pointmaps computed on the training videos. SurfSplat warns that low-resolution image metrics can hide broken geometry. So we report image quality at high resolution, and we report geometry, not image scores alone.

For allocation, VolSplat and SPLATT3RFUSION adapt primitive density using multi-view voxels or octree merging. Both depend on cross-view evidence that a single image lacks. Our allocation signal comes from uncertainty along each ray, which is available from one view.

Research Program

We build on the Flash3D recipe: a frozen UniDepth-style backbone, trained on RealEstate10K, about one GPU-day. We replace its layer head with a $K=4$ mixture head over signed depth offsets from the anchor depth. Each candidate also has footprint, opacity, and color heads. Candidates are gated by $w_{pk}$. During training we use a straight-through relaxation of the gate. At test time we apply the hard threshold $\epsilon$, which lets one trained model trade primitive count for quality.

Pseudo-depth targets for the likelihood term are VGGT pointmaps from the full training clip, aligned with the Umeyama procedure as in PM-Loss. This supervises the visible surface. Occluded layers are supervised only through rendering into other frames of the clip. After training, $\lambda$ is fitted once on a held-out split. The analysis stage asks three questions. Where does the budget go (silhouettes, glass, textureless walls)? Does learned $\sigma$ track true depth error? Does ray-aligned stretching hurt image sharpness at small camera shifts, where a stretched blob could blur what a sharp one renders correctly?

Baselines

The naive reference point is the single-layer version of our own model. It keeps one deterministic Gaussian per pixel at the anchor depth, with no mixture and no stretching, and uses the same backbone and data. It shows what the depth prior alone gives.

The strongest existing method is Flash3D. Its paper claims the best novel-view results on RealEstate10K across camera baselines, but no reported result was found on Lacuna for Flash3D's PSNR, SSIM or LPIPS. We will therefore retrain it with its released recipe, on the same data split and with the same GPU-day budget, and evaluate both models with identical code. We will run it at its native layer count and also at reduced layer counts. That gives a full curve of quality against number of Gaussians.

The allocation ablation compares against Flash3D with test-time opacity pruning down to our budget. This isolates whether learned, uncertainty-driven allocation matters, or whether discarding faint Gaussians after the fact is enough.

The shape ablation keeps our mixture and allocation but fixes ray-axis variance to the sideways footprint, giving isotropic Gaussians. This isolates the stretching.

The calibration reference is post-hoc: take the backbone's depth confidence, apply the power-law correction of the calibration audit, and stretch Flash3D's Gaussians accordingly without retraining. The audit's reported figure, 1.11x median overconfidence on held-out scenes, was measured on pointmaps, not renders. We treat it as the target level for our rendered-depth calibration, not as a directly comparable number.

Beating the baselines means two things. First, our model must match or exceed retrained Flash3D in PSNR and LPIPS at no more than half its Gaussian count. Second, its rendered 90% depth intervals must reach coverage within 10 points of nominal where the post-hoc stretch does not.

Why not just…?

Why not just prune a uniformly layered model after training? This is the most serious threat to the allocation claim. A model trained with uniform layers may already learn near-zero opacity where extra layers are useless. The opacity-pruning baseline above tests this directly at matched budgets. If it matches us, we will drop the allocation claim and present the work as uncertainty-shaped primitives only.

Why not just stretch existing Gaussians using calibrated confidence from an off-the-shelf depth model? This needs no training and uses the power-law correction that already works on pointmaps. It cannot create a second mode at a silhouette, where Revisiting Depth Representations locates the dominant failure. It also cannot move primitives, because stretching a wrong single depth only widens a wrong blob. The post-hoc baseline, evaluated separately on silhouette pixels, will show whether the mixture is needed.

Why not just initialize from monocular depth and run a short per-scene optimization? With one input image, the photometric loss is blind to depth: sliding a Gaussian along its ray leaves the input rendering unchanged. Optimization therefore cannot fix exactly the errors we target, and it adds seconds of latency per scene. We include a 500-step fit on our own initialization. We expect it to raise input-view PSNR and leave novel-view depth error unchanged, which the experiment will confirm or refute.

Why not use pixelSplat's probability-as-opacity instead? Setting opacity equal to depth probability makes uncertain pixels see-through rather than spread out, and it still spends one primitive per pixel. We implement this as a head variant on the same backbone, so the comparison is exact.

Proposed Experiments

Training and in-domain evaluation follow the RealEstate10K single-image protocol used by Flash3D. Target frames are offset by 5, 10 and random frames, and we report PSNR, SSIM and LPIPS. Following SurfSplat, we also report at higher resolution; its own two-view figures of 27.537 PSNR at standard resolution falling to 24.897 under its high-resolution protocol show how large this gap can be. Every image-quality number is plotted against Gaussian count.

Geometry and calibration are measured zero-shot on ScanNet++, which has ground-truth depth. We render mean depth and standard deviation into novel views by alpha compositing and report absolute relative depth error, coverage of the 50% and 90% intervals, and the audit's overconfidence ratio. Reliability is also split by region type: silhouettes, low texture, and reflective or transparent surfaces. Zero-shot generalization to outdoor scenes is checked on ACID. There, MonoSplat reports 28.63 PSNR, but it uses multiple views, so it is context rather than a competitor.

Three results would decide the proposal. Learned allocation must beat opacity pruning at equal budget. Ray stretching must improve coverage without costing more than 0.3 dB at the 5-frame offset. Silhouette pixels must be where the mixture's second mode is used and where depth error drops most. If allocation fails and calibration succeeds, the contribution narrows to novel-view uncertainty maps. That is still a capability no feed-forward splatting method in this literature offers.