Plane-Rectified Completion of Never-Seen Background in Distractor-Free Splatting

Methods that build 3D models of a scene from photos assume the scene holds still. When people or cars move through the capture, the best current methods treat them as noise and ignore those pixels. That works only if some other photo saw the hidden background clearly. When an object sits on the same patch of floor or road for the whole capture, such as a parked van, a lead car moving at the ego vehicle's speed, or a person standing in a hallway, nobody ever saw what was underneath. The reconstruction is left with a hole or a cloud of half-transparent blobs. The usual fix asks an image generator to paint each rendered view separately, and those paintings disagree from view to view.

We propose a different fix for the most common case, where the hidden background is a flat surface such as a road, floor, plaza, or wall. A flat surface can be "unrolled" into a single 2D texture with one cheap matrix-per-frame warp on the GPU. We warp every training photo onto that texture and count, for each texel, how many clean observations it received. A single image-generation pass then fills only the texels that were never seen, and the completed texture supervises the 3D model. Each hidden texel is generated exactly once, in one shared coordinate system, so the fill is identical from every viewpoint by construction. The warp also gives an exact map of what was truly never observed, an explicit test for disocclusion rather than a guess from rendering error.

Core Claim

For background regions hidden from every training view, filling them by generation in a plane-rectified texture space gives multi-view-consistent completions. Filling them per view in image space does not. The rectified approach should also cost one generative call per plane instead of one per view per refinement round. The homography, the $3\times 3$ matrix that maps one view of a plane onto another, is the mechanism. The static-scene violation is the regime: persistent distractors that robust reconstruction cannot see past. Disocclusion is the failure we measure.

The plane-induced homography from training frame $i$ to the rectified texture of plane $\pi$ is

$$H_i = K_\pi \left(R_i - \frac{t_i n^\top}{d}\right) K_i^{-1},$$

where $K_i, R_i, t_i$ are the intrinsics, rotation and translation of camera $i$ relative to the texture frame. The plane has unit normal $n$ and offset $d$. $K_\pi$ is a fixed fronto-parallel "virtual camera" that sets the texture resolution. Every frame is warped with GPU texture sampling. For each texel $u$ we compute a clean-observation count $c(u)$: the number of frames where the warped pixel is unmasked by the distractor detector and agrees photometrically with the robust consensus at $u$. Texels with $c(u)=0$ are the true cavity. Everything else is filled by robust temporal aggregation, using a per-texel median rather than the mean, before any generation happens.

Relation to Prior Work

The motivating gap comes from the distractor-free reconstruction line. RobustNeRF discards high-residual patches with a trimmed least-squares loss. NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features. SpotLessSplats classifies distractor pixels with diffusion-model features, because 3D Gaussian Splatting (3DGS, a scene model made of many small colored 3D blobs) produces noisy residuals during densification. RobustSplat delays Gaussian growth and bootstraps masks coarse-to-fine. All four only reject. None writes anything into a region that every view had masked. We keep their masks as an input and add the missing step.

The plane machinery comes from the pre-warping literature. Real-Time Stereo Vision for Road Surface 3-D Reconstruction warps one stereo view onto the road plane so that undamaged asphalt shows near-zero residual disparity. High Temporal Consistency through Semantic Similarity Propagation replaces optical flow with a global homography to propagate predictions at 95 fps. Deep Homography Estimation for Dynamic Scenes shows that moving objects must be masked to estimate the homography at all. These works stop once the views are aligned. The aerial-segmentation work, for instance, just down-weights its warped past prediction where things moved. We use alignment as the coordinate system for completion. Planar Gaussian Splatting shows that 3D Gaussians can be organized into planes, but it is evaluated only on static indoor scans and does no completion.

The completion side builds on decoupled warp-then-inpaint designs such as Balakrishnan et al.. It also follows the scaffolding idea of Rematas et al., where aligning a 3D CAD model turns hallucination into bounded texture-map completion. A plane is the simplest such scaffold, and it is available almost everywhere. The closest competitor is FreeFix. FreeFix renders extrapolated 3DGS views, marks uncertain pixels with Fisher-information confidence, runs an image diffusion model on each view, and folds the results back in an interleaved loop that its authors describe as slow to converge. We differ in two places. We fill in rectified texture space once instead of per view. We define "unknown" by an explicit observation count from raw images instead of by uncertainty in a model that may already contain floaters there. Heavier pipelines such as FreeOrbit4D and Beyond the Visible address object backs and edited reveals. They rely on video diffusion and, in the second case, a user who paints the hidden content. We target the automatic, static-background case.

Prior work also constrains the proposal. Split4D's and SpotLessSplats' findings that removed objects leave shadows and lighting residue (Split4D) mean the cavity cannot be defined from the object mask alone. Shadows lie on the plane, so a plane-parallax test will not flag them. This is why the observation count requires photometric agreement with the temporal median, not just mask absence. The parallax limitation of homographies means the method applies only to surfaces that pass a planarity check. Off-plane cavities fall back to a FreeFix-style per-view fill, and we report them separately.

Research Program

The system has four stages. First, run a distractor-free 3DGS baseline (RobustSplat) to obtain poses, distractor masks, and a sparse point cloud. Fit dominant planes to the static points by RANSAC, and accept a plane only if its warped observations agree across frames, measured by low residual after the warp. Second, warp all frames onto each plane's texture on the GPU. Compute the robust per-texel median and the clean count $c(u)$. Grow the cavity mask by a small margin so the generator sees a clean border. Third, inpaint the $c(u)=0$ region once with a pre-trained image diffusion inpainter, conditioned on the observed texture around it. Large driving cavities are tiled in the rectified frame, where the road has uniform scale, so the generator never sees perspective-foreshortened content. Fourth, continue 3DGS optimization with an added loss. Inside each cavity's image-space footprint, rendered pixels must match the completed texture reprojected through $H_i^{-1}$. Weight this loss below real observations and anneal it, so real data always wins where it exists. Gaussians in the cavity are initialized flat on the plane rather than by densification.

The analysis questions are where consistency comes from and where it breaks. We will vary three factors: cavity size, the fraction of the cavity that is genuinely planar, and the accuracy of the plane fit. We will also measure the seam at the boundary between real and generated texels, which the file flags as a known failure for imperfect scaffolds.

Baselines

The naive reference point is plain 3DGS trained on all pixels with no distractor handling. It shows how badly persistent occluders corrupt the scene.

The strongest rejection-only method is RobustSplat (RobustSplat). It is also our own stage one, so the comparison isolates exactly what completion adds. The file reports that it beats SpotLessSplats, WildGaussians, and T-3DGS on NeRF On-the-go and RobustNeRF scenes, but gives no per-scene number, so we re-run it with released settings. SpotLessSplats (SpotLessSplats) is run alongside it. Its reported Fountain-scene PSNR is 22.81 dB for SLS-mlp, which anchors our reproduction.

The strongest completion competitor is RobustSplat followed by FreeFix (FreeFix). We give it the same diffusion inpainter weights, the same cavity-containing test trajectories, and the same total diffusion compute measured in GPU-seconds. A second run gives it unlimited refinement rounds so it is not handicapped. No reported result was found on Lacuna for FreeFix's numbers on these benchmarks.

For driving we compare against DriveSplat (DriveSplat), which reports 34.41 dB PSNR on Waymo against 30.87 for StreetGS, and 24.53 PSNR / 0.767 SSIM on KITTI. We add our completion stage on top of DriveSplat's static background, so the gain is measured against its own output.

Two ablations isolate the key ingredient. The first, "rectified fill from 3DGS render," builds the rectified texture by rendering the RobustSplat model top-down instead of warping raw frames. It tests whether the explicit observation count matters. The second, "per-view fill, same cavity," uses our exact cavity mask but inpaints in each image independently. It tests whether rectified-space generation is what buys consistency.

Beating the field means two things. Inside cavities, we need lower held-out LPIPS than RobustSplat+FreeFix at equal compute. Across views, cross-view fill variance must be lower, as defined below. Full-image PSNR on non-cavity regions must stay within 0.1 dB of RobustSplat.

Why not just…?

Why not just run FreeFix on the holes RobustSplat leaves? This is the real threat. For small cavities seen at slight angles, per-view diffusion may agree well enough, and our method then adds nothing. The argument for our method rests on large persistent occluders. There each view's generator sees a different crop and perspective and produces different texture, and the back-projection averages the disagreements into blur, which is the multi-view inconsistency the file attributes to 2D generative fills. The per-view-fill ablation with our identical mask settles this directly. If its cross-view variance and held-out LPIPS match ours, rectification is not the cause and we would report that.

Why not just inpaint each video frame before reconstruction? Flow-based video inpainting copies background from frames where it was visible. In true cavities no frame saw it, so the generative part fills each frame independently. Beyond the Visible documents ghosting and identity loss from such stochastic fills. Flow is also the expensive component that the aerial-segmentation work replaced with a homography (Vincent et al.). We include this pipeline as a baseline on the held-out-patch protocol.

Why not just capture longer or use a stronger robust loss? For a region with $c(u)=0$ no loss can recover content, because no clean ray ever reached it. RobustNeRF even notes that its data-discarding costs it slightly on clean data. Our clean-count maps quantify how much of each benchmark scene is in this regime. If that fraction is negligible on standard benchmarks, we will say so and rest the case on the driving and planted-occluder settings.

Why not just render the reconstructed plane top-down and inpaint that? In a cavity the 3DGS model contains floaters or nothing. Its render therefore mixes garbage with real texture, and nothing marks which is which. The "rectified fill from 3DGS render" ablation measures this. If it ties, the raw-frame homography warp is merely a speed convenience and we would describe it that way.

Proposed Experiments

Hallucinated content has no ground truth, so the decisive check uses a planted-occluder protocol. On the RobustNeRF benchmark's paired captures, where clean background exists and RobustNeRF reports 30.87 dB versus 25.22 for Mip-NeRF 360, we project a 3D footprint on the floor plane into every training image and mask it in all of them. We then evaluate masked PSNR, SSIM, and LPIPS at test views where the true floor is visible. We repeat this on Waymo and KITTI with footprints placed on the road under simulated lead vehicles. On real persistent occluders, such as same-speed vehicles in Waymo and standing people in NeRF On-the-go scenes, there is no ground truth. For those we report cross-view fill variance: render the cavity from 20 test poses, warp each render back to the rectified texture, and take the per-texel standard deviation. We also report temporal flicker along a smooth trajectory and a blind pairwise preference study against RobustSplat+FreeFix.

Secondary checks cover three things. One is shadow residue: on-plane photometric deviation near removed objects in On-the-go scenes, where the On-the-go paper reports 21.41 dB on Patio, compared with mask-based baselines. Another is the seam-error profile as a function of plane-fit error. The last is wall-clock and diffusion GPU-seconds per scene.

The proposal fails if the per-view-fill ablation matches us on consistency. It also fails if planted-occluder LPIPS does not improve over RobustSplat+FreeFix at equal compute. Either outcome would tell us that the plane is only a speedup, not a better completion space.