Feed-forward 3D reconstruction trains one network to turn a handful of photos into a renderable 3D scene in a fraction of a second. This removes the minutes-to-hours of per-scene fitting that classic methods need. Most such networks place 3D primitives (small colored Gaussian blobs, the "splats" of 3D Gaussian Splatting) by guessing one depth per pixel and pushing a blob out along that pixel's ray. Depth from a few images is often ambiguous: many 3D arrangements produce the same picture, especially with little view overlap or on textureless surfaces. In those cases the single guess is often wrong, and every structure built afterwards inherits the mistake. That includes the octrees recently used to compress these scenes. An octree is a tree that recursively splits a box into eight smaller boxes only where detail exists.
We propose to build the octree from the network's depth distributions rather than its depth guesses. The tree is fine where several views agree on a surface. It stays coarse, keeping several depth hypotheses alive, where depth is genuinely uncertain. A few gradient-free feedback steps then re-split or merge cells as rendering error from the input views resolves the ambiguity. In plain terms, the tree's resolution follows how sure the model is about where things are, after that confidence has been corrected for known overconfidence. Wrong early pruning, the main failure of tree-building from images, becomes recoverable.
Core Claim
Existing feed-forward octrees decide space allocation after depth has already been collapsed to a point. Learning Compact 3D Gaussians via Feed-Forward Point Fusion merges VGGT-predicted points inside octree cells by feature similarity. Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting deposits a base model's Gaussians into a voxel hierarchy and keeps the top 20% of voxels. Both are post-processors. When the upstream depth is wrong, points from different surfaces land in the same cell or scatter, and nothing downstream can move them across large distances.
Our claim is that, on inputs where depth is ambiguous, moving the octree before depth commitment buys something specific. It should produce fewer floaters and better geometry and view synthesis at a matched primitive and memory budget. The tree would be driven by calibrated per-ray depth distributions and revised by error feedback. On easy, well-overlapped inputs we expect parity with post-hoc fusion, and we treat that parity as an honest prediction rather than a failure.
The signal that drives subdivision is a cell's accumulated, calibrated depth evidence. Each context view $v$ predicts, for pixel $p$, a probability $\pi_{v,p,k}$ over $K$ depth buckets, in the style of the bucketed depth used in pixelSplat. Bucket $k$ corresponds to the 3D point $x_{v,p,k}$ on that pixel's ray. A cell $c$ collects evidence
$$ m(c) = \sum_{v,p,k} g(\pi_{v,p,k})\,\mathbb{1}[x_{v,p,k} \in c], $$
where $g$ is a recalibration map fitted on held-out scenes.
A cell splits when its evidence, normalized by the number of rays passing through it, exceeds a threshold and its projected footprint in some input view exceeds a pixel budget. Rays that agree across views concentrate mass where they intersect, the ray-intersection idea used for multi-agent perception in RayFusion (Wang), and those cells subdivide. A single ambiguous ray spreads its mass thinly, so its cells stay coarse and keep a latent "hypothesis" feature instead of being pruned.
The recalibration $g$ is not optional. A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction found feed-forward backbones underestimate geometric error by a median 2.4×. It found MASt3R's nominal 90% intervals cover the truth only 0.6% of the time. It also found a power-law recalibration reduces overconfidence to 1.11× on held-out scenes. A tree split on raw confidence would therefore prune precisely the regions where the model is confidently wrong. We reuse that recalibration as a component.
Research Program
The system has three stages trained end to end with rendering losses only. It uses no LiDAR or occupancy ground truth, unlike OctOcc and OctreeOcc. Both of those show octrees can tame cubic memory for camera-based occupancy, but both target semantic voxels and rely on LiDAR-derived supervision.
Stage 1: depth distributions. A 2D encoder produces per-pixel bucketed depth distributions. Following DepthSplat, it fuses a plane-sweep stereo branch with Depth Anything V2 features. DepthSplat uses this to pick one depth per pixel and is strictly pixel-aligned; we keep the whole distribution.
Stage 2: tree construction and Gaussian decoding. The octree is built by the evidence rule above, from a coarse root to a maximum depth. Image features are gathered into each active cell by attention restricted to the rays that pass through it. This is a hierarchical, axis-aligned generalization of the spherical anchor-to-ray attention in LocusGS, whose scalar radius cannot adapt to thin or planar structures and whose anchors form a flat set rather than a hierarchy. Leaves decode Gaussians as offsets inside their bounds. Coarse hypothesis cells decode low-opacity, large Gaussians, echoing the coarse "skeleton" Gaussians of HiSplat. HiSplat's hierarchy is over primitive scale only and never partitions or prunes space.
Stage 3: error-driven tree revision. Over $T$ recurrent steps, with $T=4$ as in ReSplat, the current Gaussians are rendered into the input views. Per-pixel photometric and feature error is scattered back to the cells whose Gaussians contribute to each pixel. A small update network then does three things: it adjusts Gaussian parameters, adds evidence to cells that explain error well, and removes evidence from cells that do not. The split rule is then re-applied, so cells can subdivide, collapse, or be revived from a coarse parent.
ReSplat showed this kind of gradient-free feedback beats 4,000-iteration 3DGS in four steps. However, it retrieves neighbors by flat Euclidean k-nearest-neighbor search over all Gaussians, which it identifies as a scaling bottleneck. Here the tree supplies neighborhoods and confines updates to cells touched by high error.
The analytical deliverable is a measured account of where the tree's resolution goes. We will report cell depth versus recalibrated depth uncertainty, and cell depth versus rendering error. We will also report how often a region pruned or kept coarse at step 0 is correctly resolved by step $T$. This "recovery rate" directly quantifies the chicken-and-egg problem that hierarchical partitioning from 2D observations has always faced.
Baselines
Naive reference. The do-nothing point is DepthSplat run as published: pixel-aligned, single pass, one Gaussian per pixel per view. It shares our depth encoder, so any gain is attributable to what happens after depth prediction. We will retrain it on our exact data splits and view counts. No absolute DepthSplat PSNR is reported in the file; the file records only that it improves on MVSplat by nearly 2 dB on DL3DV and runs in 0.6 s for 12 views at 512×960.
Strongest view-synthesis competitor at matched cost. This is VolSplat, which unprojects features into a uniform voxel grid, refines them with a sparse 3D U-Net, and emits Gaussians only from occupied voxels. It reports 31.30 dB PSNR, 0.941 SSIM and 0.075 LPIPS on RealEstate10K with 6 views, at 0.575 s and 4.65 GB VRAM on an H20. It is the most direct test of whether adaptive, uncertainty-shaped subdivision beats uniform voxelization. We will run it on the same views and resolution and give it the same encoder where its code allows. We will also give it a second configuration whose voxel size matches our finest leaf, to expose the memory cost of uniform resolution. Beating it means matching or exceeding 31.30 dB at equal or lower VRAM on RE10K 6-view. It also means a clear margin on our high-ambiguity split and at high resolution, where its uniform grid must coarsen to fit memory.
Closest structural competitor. This is post-hoc octree fusion as in Learning Compact 3D Gaussians via Feed-Forward Point Fusion. It reports 40.0k Gaussians versus NoPoSplat's 131.1k on RE10K at τ=0.995, and matches Splatt3R on ScanNet++ with 15.5% of the primitives. No absolute PSNR is reported in the file. We will also include Fuse-and-Refine on top of DepthSplat, which reports +2.37 dB over GS-LRM with a 15 ms refiner. Both will be applied to the same DepthSplat backbone and swept over their compaction thresholds. That lets us compare full PSNR-versus-primitive-count curves rather than single points.
Strongest refinement competitor. This is ReSplat with the same $T$. It reports outperforming DepthSplat by 3.5 dB on DL3DV 8-view with 246K Gaussians; no absolute PSNR is reported in the file. On DL3DV 4-view at a fixed 262K-Gaussian budget, LocusGS reports 24.80 dB PSNR, which we will match on budget.
Ablations. These isolate the key ingredient. The first builds the tree from the argmax depth instead of distributions, which collapses to point fusion inside our own architecture. The second uses raw instead of recalibrated evidence. The third disables tree revision, keeping the step-0 tree fixed across the $T$ updates. The fourth replaces the octree with a uniform grid of the finest leaf size wherever memory allows.
Why not just…?
Why not just run post-hoc octree fusion on a strong depth backbone? This is the most serious threat. On scenes with good overlap, depth is nearly unambiguous, so collapsing to points loses little; we expect our method to tie here. The case for our design rests on ambiguous inputs, where Learning Compact 3D Gaussians via Feed-Forward Point Fusion itself depends on correct VGGT coordinates. Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting shows that pixel-aligned depth already fragments at object boundaries badly enough that a pointmap loss gains at least 2 dB. The argmax-tree ablation settles this directly. If it matches the distribution-driven tree on the high-ambiguity split, the core claim is false and we will say so.
Why not just soft-voxelize depth distributions into a uniform grid, VolSplat-style? This captures the hypothesis-keeping benefit without any tree. It will likely work at moderate resolution. Our claimed advantage is memory and resolution in deep or large scenes, where uniform grids hit the cubic wall. OctOcc reports this wall concretely: a dense 200×200×16 grid exceeds 32 GB, while its octree fits in 23.4 GB. The uniform-grid ablation run at matched VRAM decides the question. If uniform soft voxels win at every budget we can fit, the octree is not earning its irregular-memory cost.
Why not just add a few hundred iterations of test-time optimization, or more ReSplat steps? Optimization can move a misplaced surface. ReSplat's evidence that four learned steps beat 4,000 gradient iterations suggests the bottleneck is not refinement capacity. We will nevertheless report a depth-initialized 3DGS run at 500 and 1,000 iterations, with wall-clock time. We will also report ReSplat at $T=8$, to check whether extra flat steps close the gap our tree revision claims to close.
Why not just prune with the backbone's own confidence? The calibration audit answers this empirically: raw confidence is most overconfident exactly where it is highest. The raw-evidence ablation measures how much true surface such pruning discards, by comparing against ground-truth depth on ScanNet++ and DTU.
Proposed Experiments
We train on RealEstate10K and DL3DV in their 2-, 4-, 6- and 8-view settings, and evaluate in-domain with PSNR, SSIM and LPIPS. Every result also reports Gaussian count, peak VRAM and latency on one GPU type. Zero-shot transfer is tested on Replica, where HiSplat reports +3.19 dB over prior feed-forward methods; on ACID, for outdoor generalization; and on DTU and ScanNet++, where we report Chamfer distance, accuracy and completeness against ground-truth geometry.
The decisive experiment stratifies every test set by ambiguity. We compute the fraction of each target view's pixels co-visible in at least two context views, plus the recalibrated depth entropy, then bin scenes into terciles. The prediction is specific. Over the argmax-tree ablation, post-hoc fusion and VolSplat at matched primitive count, the gain should grow from roughly zero in the low-ambiguity tercile to a clear margin in the high-ambiguity tercile. The margin should show up in both PSNR and Chamfer distance. A flat gain profile would mean the benefit comes from elsewhere, not from handling ambiguity.
We will also report the recovery rate: the fraction of ground-truth surface voxels absent or coarse at step 0 that become correctly resolved leaves by step $T$. We expect it to be substantially higher with revision enabled. Finally, we measure the calibration of the evidence itself, as coverage of recalibrated depth intervals in the style of the audit. This checks that the quantity steering the tree means what we claim it means.
References
- Learning Compact 3D Gaussians via Feed-Forward Point Fusion
- Learning Efficient Fuse-and-Refine for Feed-Forward 3D Gaussian Splatting
- PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction
- RayFusion: Ray Fusion Enhanced Collaborative Visual Perception
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree
- OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries
- DepthSplat: Connecting Gaussian Splatting and Depth
- LocusGS: Spatially Grounded Tokens for Feed-Forward 3D Gaussian Splatting
- HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction
- ReSplat: Learning Recurrent Gaussian Splats
- VolSplat: Rethinking Feed-Forward 3D Gaussian Splatting with Voxel-Aligned Prediction
- Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting