Feed-forward 3D reconstruction models predict a full 3D scene from a few photos in one network pass, with no per-scene optimization. They usually represent the scene as thousands of small colored 3D blobs, called Gaussian splats. The obvious way to make them accurate is to train them on synthetic scenes, where exact depth and camera poses come for free. The cost is that networks trained on rendered images pick up rendering-specific texture cues and then fail on real photos. We propose a narrower route. Many deployed systems already have a sparse geometric sensor: a LiDAR, a depth camera, or the sparse 3D points a visual-inertial tracker produces. We train the part of the reconstructor that decides where the Gaussians go only on synthetic sparse-depth-to-dense-geometry pairs, and that part never sees an image. A second, image-driven network then corrects that geometry, and predicts color, using only real video and a self-supervised rendering loss.
The bet is that the synthetic-to-real gap lives mostly in appearance, while the mapping from a scattered set of 3D points to a dense surface looks the same in simulation and in reality. If that holds, a color-blind geometry module should lose much less accuracy when moved from synthetic to real data than an equally sized module that also sees synthetic images. A second consequence is more useful in practice. Because the color-blind module's inputs look alike in both domains, an uncertainty estimate calibrated on synthetic data should stay calibrated on real scenes. That would address an open problem identified by a recent audit of feed-forward confidence scores, which asks how to set calibration without real labels.
Core Claim
The principle comes from Learning Topology from Synthetic Data for Unsupervised Depth Completion. That paper trains a small network (ScaffNet) on synthetic sparse depth alone to produce a dense depth "scaffold." A second network (FusionNet) refines the scaffold with real RGB through a per-pixel scale and offset, and is trained without labels on real images. It reports a mean absolute error of 280.76 on KITTI, against 299.41 for the previous best. That work produces only 2.5D depth maps, cannot render new views, and uses a single view.
We carry the principle into multi-view feed-forward Gaussian splatting. The color-blind scaffold predicts Gaussian centers and a per-Gaussian uncertainty from multi-view sparse depth and camera geometry. An RGB fusion network then predicts a bounded correction to each center along its camera ray, together with the Gaussian's shape, opacity, and color. For a pixel ray with scaffold depth $d_0$ and scaffold uncertainty $\sigma_0$, the final center depth is
$$d = \alpha\, d_0 + \beta, \qquad |\alpha - 1| \le \epsilon,\; |\beta| \le k\,\sigma_0,$$
where $\alpha$ and $\beta$ are the fusion network's predicted scale and offset, $\epsilon$ is a small fixed scale tolerance, and $k$ sets how many scaffold standard deviations the image may move the surface.
This bound is the point of the design. The image may sharpen boundaries and fill regions the sparse sensor missed, but only within the scaffold's own error bars. Real-world texture therefore cannot drag the geometry far from what the sensor supports.
The claim is falsifiable. If a coupled network trained on mixed synthetic and real RGB-D data matches the color-blind scaffold on real geometry and calibration, decoupling buys nothing and the proposal is wrong.
Relation to Prior Work
The closest existing system is MapAnything. It already accepts RGB together with optional depth, poses, and intrinsics, and trains on 13 synthetic and real datasets. Its geometric inputs pass through shallow encoders and are concatenated with DINOv2 image features early in the network. Its synthetic images therefore shape the same features that place geometry, and it models no noise in the geometric inputs. We differ in two ways: our geometry pathway is never exposed to synthetic pixels, and its uncertainty explicitly limits how far the image pathway can move the geometry. MapAnything is the competitor this proposal must beat. It also constrains the design, since it shows that heavy mixed-domain training is already a strong recipe.
Three methods supply the feed-forward machinery we reuse rather than reinvent. MVSplat builds a plane-sweep cost volume, which scores how well image features match across views at a set of candidate depths, and predicts Gaussians from it. FreeSplat adds a learned merging step that removes duplicate Gaussians across overlapping indoor views. Learning Compact 3D Gaussians via Feed-Forward Point Fusion merges points into octree cells and reaches parity with Splatt3R using 15.5% of the primitives. All three take RGB only and contain no mechanism against domain shift. Two of them also note that matching fails on textureless and reflective surfaces, which is exactly where a sparse depth sensor helps.
Four papers motivate the proposal by documenting the gap. ITSA shows that stereo networks trained on synthetic Scene Flow learn shortcuts: on KITTI 2015, D1 error is 29.3% without its shortcut penalty and 5.8% with it. S2R-DepthNet removes style by training structure extraction on paintings. A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction finds that feed-forward backbones underestimate their error by a median factor of 2.4×, and that MASt3R's nominal 90% intervals cover the truth only 0.6% of the time. Its dataset-specific power-law fix brings the overconfidence factor to 1.11× but requires labeled data from the target domain, and it lists label-free calibration as open. Our calibration-transfer test is aimed at that open problem.
Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting regularizes Gaussian centers toward VGGT pointmaps and reports gains of at least 2 dB. It is both a method we can borrow and a competitor, discussed below. SGFormer supplies the fusion operator. In it, 3D tokens query image tokens through cross-attention, which beats concatenation by 2–10% panoptic quality under sensor disturbance. We use the same direction of attention, with geometry querying the image. Countering Multi-modal Representation Collapse through Rank-targeted Fusion warns that a dense modality can drown out a sparse one. Our bounded correction is a structural guard against that failure.
Research Program
The work has three components, each tested separately.
The color-blind scaffold. Its inputs are, for each view, a sparse depth map with a validity mask, plus camera intrinsics and relative poses. It warps the sparse depth points of all views into a shared cost volume indexed by depth plane, so it is still multi-view, but it matches geometry rather than texture. It outputs per-pixel depth, a log-variance, and an occupancy signal used to drop redundant Gaussians in overlapping views.
It is trained only on synthetic indoor and outdoor scenes drawn from datasets named in the file: Aria Synthetic Environments, TartanAirV2, Virtual KITTI 2, and Replica. Sparse inputs are sampled to imitate real sensors: LiDAR-like scan lines for driving scenes, feature-point sampling at about 0.05% density for visual-inertial input (the regime where Learning Topology reports robustness on VOID), and depth-camera dropout on dark and specular regions. Depth and uncertainty are supervised with a Gaussian negative log-likelihood against exact synthetic depth.
Simulating real sensor noise is the weakest link in the design, since real sensor noise is not texture-free. We treat it as an explicit variable. We train separate scaffolds under clean, moderate, and aggressive noise models, and report how real-domain error depends on the choice.
The RGB fusion network. The image backbone is MVSplat-style. Scaffold tokens query image features through cross-attention, and the network outputs the bounded correction above together with covariance, opacity, and color. It is trained only on real RGB-D or RGB-plus-sparse-point video: ScanNet and ScanNet++ indoors, KITTI outdoors. The loss is photometric rendering of held-out frames, the same signal NoPoSplat and MVSplat use, with no real depth labels. The scaffold stays frozen during this stage, which is what keeps it color-blind.
Calibration transfer. We fit the audit's two-constant power-law recalibration to scaffold uncertainty on synthetic validation scenes only. We then measure interval coverage on real scenes with no refitting. We repeat this for a coupled scaffold and for VGGT's native confidence.
Baselines
The naive reference point has two parts. The first is MVSplat trained on the same real ScanNet data with RGB only, which measures what the sensor adds at all. The second is a no-learning baseline that densifies the sparse depth by nearest-neighbor interpolation, unprojects it to Gaussians, and fits colors with the same fusion head. This shows how much of any gain comes from simply having metric points. For reference, FreeSplat reports PSNR 28.08 for 2-view and 27.45 for 3-view novel view synthesis on ScanNet. MVSplat reports PSNR 26.39 on RealEstate10K. Ours must at least match the ScanNet figure with RGB-only inputs to show the image pathway is not handicapped.
The strongest existing competitor is MapAnything given RGB plus the same sparse depth. It is strongest because it already fuses these modalities and was trained on far more mixed synthetic and real data than we will use. We will run its released model and attach the same Gaussian-prediction head. That head is trained on the same real ScanNet/KITTI clips, with the same number of steps and the same hyperparameter search as our fusion network, so the only difference is where geometry comes from. No reported result found on Lacuna for MapAnything on our metrics, so we will measure it ourselves. Beating it means lower depth absolute relative error and Chamfer distance on ScanNet++ and KITTI with equal or better PSNR. Separately, beating it on calibration means real-domain 90% interval coverage within five points of nominal, without real-domain fitting.
A second strong competitor is MVSplat with sparse depth concatenated as an extra input channel, trained end-to-end on the same data. This represents the early-fusion design. A third is MVSplat with VGGT-pointmap regularization following Revisiting Depth Representations. It uses no sensor but brings a large foundation-model prior. On depth completion specifically, we compare to the original FusionNet's reported KITTI mean absolute error of 280.76, which our per-view depth must not exceed.
Two ablations test the key ingredient. The decisive one is the coupled scaffold: identical architecture, synthetic data, and compute, except the scaffold also receives synthetic RGB, and a variant trains it on mixed synthetic plus real RGB-D. The second removes the bound on $\beta$, so the image can move geometry freely. That isolates whether the uncertainty-limited correction prevents the image modality from overpowering the sparse geometry.
Why not just…?
Why not just use MapAnything with depth inputs? This is the real threat, and we do not claim to know the answer. MapAnything's scale may already have pushed shortcut learning below what matters. The coupled-scaffold ablation, trained on mixed synthetic and real data, imitates its recipe at matched size. The MapAnything comparison tests the full-scale version. If either matches the color-blind scaffold on real geometry, we will report that decoupling is unnecessary at scale. In that case the remaining question is whether only the calibration-transfer result survives, which MapAnything cannot be tested on because it predicts no uncertainty.
Why not just train on real video with photometric loss and skip synthetic data? That is what NoPoSplat and MVSplat do. The file's evidence is that photometric-only training leaves boundary fragmentation, which PM-Loss repairs for at least 2 dB, and floaters. Transient objects also hurt it: in Distractor-free Generalizable 3D Gaussian Splatting, MVSplat reaches PSNR 15.45 on the RobustNeRF "Statue" scene versus 21.02 with distractor handling. Photometric loss also gives no supervision for uncertainty. The concatenated-depth MVSplat baseline is exactly this alternative with the sensor added, so the comparison settles it directly.
Why not just apply shortcut penalties or style randomization to a coupled model? ITSA and S2R-DepthNet show these help: S2R-DepthNet reaches KITTI absolute relative error 0.165 with zero real images. But it still trails semi-supervised training at 0.116, and both methods still let texture reach the geometry features. We will add ITSA's Fisher-information penalty to the coupled scaffold as an extra arm. If it closes the gap, the cheaper fix wins on geometry, and our contribution narrows to calibration transfer.
Why not just use a foundation model's pointmaps? VGGT-style priors need no extra sensor, which is a real advantage. However, the audit shows these backbones get relative shape right while misjudging metric scale per scene, and that they are badly overconfident. Errors in the pointmap pass directly into PM-Loss geometry. The PM-Loss baseline measures this. We expect it to compete on PSNR but lose on metric depth and coverage.
Decisive Checks
The first check is the transfer ratio. For each geometry module, divide real-domain depth error on ScanNet++ and KITTI by its error on held-out synthetic scenes. The proposal predicts a clearly smaller ratio for the color-blind scaffold than for the coupled scaffold, at matched synthetic accuracy. If the ratios are equal, the core claim fails.
The second check is calibration transfer. Fit the recalibration on synthetic data, then measure real 90% interval coverage. We expect the color-blind scaffold to approach nominal coverage. We expect the coupled scaffold and VGGT confidence to stay overconfident, closer to the audit's 2.4× regime.
The third check is a sensor-density and noise-model sweep, from 0.05% density to dense depth and from clean to aggressive simulated noise. It finds where the scaffold's advantage disappears and how sensitive transfer is to the sensor simulation.
The fourth check is the unbounded-correction ablation on textureless and reflective ScanNet regions, where we expect image-driven corrections to do the most damage.
Together these checks separate three outcomes: decoupling helps geometry, decoupling helps only calibration, or decoupling helps neither. Each outcome is a reportable answer.
References
- Learning Topology From Synthetic Data for Unsupervised Depth Completion
- MapAnything: Universal Feed-Forward Metric 3D Reconstruction
- MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-view Images
- FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes
- Learning Compact 3D Gaussians via Feed-Forward Point Fusion
- ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks
- S2R-DepthNet: Learning a Generalizable Depth-Specific Structural Representation
- Revisiting Depth Representations for Feed-Forward 3D Gaussian Splatting
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation
- Countering Multi-modal Representation Collapse through Rank-targeted Fusion
- No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images
- Distractor-free Generalizable 3D Gaussian Splatting