Bounded Deformable Cost Volumes for Sim-to-Real Feed-Forward 3D Reconstruction

Feed-forward 3D reconstruction networks look at a few photos and, in one pass of a neural network, output a full 3D scene that can be rendered from new viewpoints in milliseconds. The scene is stored as thousands of small colored blobs called 3D Gaussians. The strongest of these networks find depth by checking, for every pixel and every candidate depth, whether the features at the matching location in another photo agree. That matching location is computed exactly from the known camera poses. Synthetic training data is cheap and comes with perfect poses, perfect synchronization and a global shutter. A network trained on it therefore learns to trust that exact geometric correspondence completely. Real cameras break this trust. Poses are estimated with error, rolling shutters skew each row slightly in time, and frames are blurred. Matching then lands a few pixels off, and the depth estimate degrades.

We propose to let each depth-matching lookup search a small, learned, strictly bounded neighborhood around its geometric target. The idea comes from deformable convolution, which predicts per-location sampling offsets. We compute those offsets once per view pair rather than iteratively, so reconstruction latency stays close to that of a rigid network. We train this "deformable cost volume" on synthetic scenes into which we inject realistic geometric corruption: pose error, rolling-shutter skew and timing offsets. The central claim is that a measurable part of the synthetic-to-real gap in feed-forward reconstruction is geometric rather than photometric. Data augmentation alone cannot close that part in a rigid network, while bounded learned offsets can, at negligible cost in speed. The claim fails if augmentation alone, or a global pose-correction head, does just as well. The experiments are designed to test exactly that.

Core Claim and What Is New

A cost volume is a table of matching scores between a reference image and a source image, one score per pixel and per candidate depth. MVSplat builds this table by plane-sweeping: each depth hypothesis is projected into the other view and features are compared at that exact point. Gaussian centers come from the resulting depth, and opacity comes from matching confidence. FreeSplat extends the same rigid construction to longer indoor sequences. It adds a recurrent fusion module that removes duplicate Gaussians, and it reports zero-shot transfer from ScanNet to the synthetic Replica dataset. Both papers supply our base architecture, and both assume the projection is correct. We relax that assumption. We replace the matching score at reference pixel $p$ and depth $d$ with

$$ C(p,d) = \sum_{k=1}^{K} m_k(p,d)\,\big\langle F_r(p),\, F_s\big(\pi(p,d) + \Delta_k(p,d)\big)\big\rangle, \qquad \|\Delta_k\|_\infty \le r . $$

Here $F_r$ and $F_s$ are the reference and source feature maps, and $\pi(p,d)$ is the rigid projection of pixel $p$ at depth $d$ into the source view. $\Delta_k$ are $K$ learned 2D offsets, clipped to a radius of $r$ pixels by a scaled tanh. $m_k \in [0,1]$ are learned gates that let the network ignore occluded or saturated samples. A small convolutional head predicts the offsets and gates in a single shot from the reference features and the coarse rigid cost.

The bound $r$ matters. It keeps the epipolar constraint, which MVSplat shows is what suppresses floating artifacts. It also stays inside the receptive-field regime where DenseWarper found learned offsets converge; that work reports failure when displacements grow too large.

The closest mechanism is GADA. It adds recurrent deformable offsets, five iterations, to correct reprojection errors in image-based Gaussian splatting. It raises valid pixel density from about 33% to about 79% compared with rigid visibility checks. GADA motivates our design because it shows rigid projection discards most usable correspondences when geometry is imperfect. Our setting differs in three ways. GADA uses offsets at rendering time over an optimized scene. We place them inside a generalizable reconstruction network's depth search, compute them in one shot, and use them as a deliberate tool for transferring from synthetic to real data. That use is untested in GADA, pixelSplat, MVSplat and FreeSplat.

The domain-gap motivation comes from three papers. ASM shows that networks exploit the difference between clean synthetic and noisy real inputs as a shortcut. Its synthetic-trained LiDAR segmentation reaches 22.8% mIoU against a 55.5% real-data oracle even after adaptation. Driving in the Matrix and FAINT show the gap can close for 2D detection and navigation, through synthetic scale and through frozen pretrained encoders respectively. None of the three involves dense multi-view geometry, which is where we expect the shortcut to be geometric.

Research Program

We build on the MVSplat codebase and swap only the cost-volume construction. Everything else is kept fixed: encoder, Gaussian head, renderer and loss. For synthetic training data we render multi-view clips from Replica and HM3D-style indoor scenes. Renders use perfect poses and a global shutter by default.

The corruption model has three parameterized families. The first is pose error: rotation and translation noise matched to typical structure-from-motion error. The second is rolling-shutter skew: each row is rendered at a slightly different camera time along the trajectory. The third is inter-frame timing offset, a small temporal stagger between views in the style of DenseWarper. Photometric augmentation (noise, blur, exposure) is applied identically to every model we compare, so any difference we measure comes from geometry.

The analysis has three parts. First, we measure how much offset each model actually uses on synthetic test data with known injected corruption. If the offsets are doing what we claim, their magnitude and direction should track the injected displacement field. On real data, offsets should grow with image row under fast camera motion, as a rolling shutter would predict. Second, we sweep $r$ and $K$ to locate the point where offsets start to erode geometric precision. We measure this directly with depth error against ScanNet's sensor depth, because BTimer notes that pixel-aligned predictors can look sharp while being metrically wrong. Third, we profile latency. Gaussians are output exactly as before, so rendering speed is unchanged. The only added cost is the offset head and $K$ samples per cost entry. We target at most 15% over the rigid network's reconstruction time.

Baselines

The naive reference point is MVSplat with its rigid plane-sweep cost volume, trained on the synthetic data with photometric augmentation only, then tested zero-shot on real data. This defines the size of the gap. There is no reported result on Lacuna for a synthetic-trained feed-forward Gaussian reconstructor evaluated on ScanNet or RealEstate10K, so we must measure this number ourselves.

The strongest competitor on the central question is the same rigid MVSplat trained with our full geometric corruption model. It isolates the question that matters most: do the offsets add anything beyond showing the network jittered data? It uses the identical data stream, number of iterations and hyperparameter search budget. We give it the same number of tuning runs, including a variant with a wider set of depth hypotheses, to compensate for its lack of spatial search.

We bound performance from above with real-data oracles. FreeSplat reports 28.08 dB PSNR with 2 views and 27.45 dB with 3 views on ScanNet when trained on ScanNet. MVSplat reports 26.39 dB PSNR on RealEstate10K with 44 ms per frame. We retrain both on real data in our setup, so the oracle uses the same resolution and view counts as our models.

Beating the field means two things on ScanNet and RealEstate10K. First, the deformable model must close a clearly larger share of the gap between the naive synthetic model and the real oracle than rigid-plus-augmentation does. We set the bar in advance at one third more of the remaining gap. Second, it must not worsen depth error and must stay within the latency budget.

Four further variants share the same training and isolate the key ingredient. The first is a feed-forward pose-correction head: a rigid cost volume with a predicted 6-degree-of-freedom correction per source view. The second is optical-flow pre-warping. The third is the deformable model trained without geometric corruption, to test whether offsets alone help. The fourth is an unbounded, five-iteration recurrent version in the style of GADA, to test whether our bound and single shot cost accuracy.

Why not just…?

Why not just train the rigid network with geometric augmentation? This is the most serious threat, and it is why rigid-plus-augmentation is our main baseline rather than an afterthought. A rigid cost volume can respond to jittered training only by blurring its matching scores, through wider confidence and lower opacity. It cannot move its samples to where the true match lies. We therefore expect it to gain robustness at the price of sharpness. GADA's finding that rigid checks keep only about 33% of valid pixels against about 79% with offsets suggests the same trade-off. If rigid-plus-augmentation matches our model on real-data PSNR and depth error, we will report that the gap is a data problem, not an architecture problem.

Why not just predict a corrected camera pose and keep the cost volume rigid? A pose correction fixes errors that are the same everywhere in the image, but not rolling-shutter skew or per-region timing offsets, which vary row by row. Our corruption sweep separates the two cases. With pose noise alone, the pose head should roughly tie our model. With rolling shutter it should fall behind. If it ties on real data too, the real gap is mostly global pose error, and the simpler head wins.

Why not just warp neighboring frames with optical flow first? Flow estimation assumes brightness stays constant between frames, and it adds a separate heavy network. The deformable-alignment literature summarized in LIF-VSR avoids flow for this reason and reports +1.32 dB on Vid4 from implicit deformable alignment over an unaligned baseline at 17.60 GFLOPs. More fundamentally, flow warps toward 2D motion rather than toward a depth hypothesis, so it can conflict with the epipolar constraint. We include flow pre-warping as a measured baseline and report both accuracy and its added latency.

Why not just freeze a foundation encoder such as DINOv2, as FAINT did? FAINT's 90% real navigation success shows frozen features transfer well in appearance. That addresses the photometric gap, which we deliberately hold fixed, rather than misaligned geometry. Such encoders also produce features on a coarse patch grid, typically one feature per 14 or 16 pixels, too coarse for the sub-pixel matching that Gaussian placement needs. ARTDECO, which relies on the foundation geometry model MASt3R, reports that its priors fail under distribution shift. The two ideas are complementary, so we also test a frozen-encoder variant with and without offsets. That shows whether the geometric gain survives once the appearance gap is reduced.

Proposed Experiments and Decisive Checks

The first experiment is a controlled synthetic sweep on held-out Replica and HM3D scenes. Each corruption family is varied separately across severity levels. We plot PSNR, SSIM, LPIPS and depth error against severity for every model. The prediction is that our model's curve stays flat out to about the bound $r$, while the rigid models degrade.

The second experiment is zero-shot synthetic-to-real transfer to ScanNet, for 2- and 3-view inputs and for ScanNet's sensor depth, and to RealEstate10K, whose YouTube-sourced, structure-from-motion-posed clips naturally contain rolling shutter and pose error. We report the fraction of the gap closed relative to the real-trained oracles.

The third is a latency audit: reconstruction milliseconds and rendering frames per second against MVSplat's 44 ms.

The decisive check is the offset diagnostic on real footage. If the learned offsets on RealEstate10K correlate with image row under fast camera motion and with known pose residuals, the offsets are absorbing geometric error. If they look like noise while PSNR still improves, the gain more likely comes from extra model capacity. We will report that honestly, rather than claim a geometric mechanism.