Learning Hidden Walls in Simulation, Painting Them from Real Images

When a camera moves, it reveals surfaces that were hidden behind foreground objects. This is called disocclusion, and the revealed regions have no pixels in the source image, so any view-synthesis system must guess what is there. In indoor scenes most of what gets revealed is a wall, floor, ceiling, or tabletop continuing behind a sofa or a chair. The hard part of that guess is geometric: where does the wall behind the sofa end, and where does it meet the floor? Synthetic 3D worlds answer this question perfectly, because every hidden surface is known. Models trained on synthetic renderings, however, degrade on real photos (the sim-to-real gap), and they degrade mostly because of appearance: clean textures, perfect lighting, no sensor noise.

We propose to split the problem along this fault line. A small network, trained only on synthetic indoor scenes, predicts the hidden planar layout behind occluders. That layout is the set of planes (each just a normal direction and an offset) together with their extents and intersection lines in the occluded region. The texture of each revealed plane is then filled by a frozen image inpainter pretrained on real photographs, working in a view where that plane faces the camera head-on and its texture looks like an ordinary flat image. The bet is that plane parameters and layout reasoning are nearly domain-invariant, while texture is not. The simulator should therefore teach only geometry, and the real-image prior should supply only appearance.

Core Claim

Our claim is that, for wide-baseline indoor view synthesis from one or a few real views, a synthetic-trained amodal planar layout predictor combined with a real-trained, plane-rectified texture inpainter will produce more accurate disoccluded geometry and fewer artifacts than three alternatives. The first is end-to-end sim-to-real view synthesis. The second is generative RGB-D inpainting. The third is simple geometric extrapolation of visible planes. The gain should concentrate in the regions that are actually disoccluded, not in the visible regions.

Prior work supplies the pieces but not this decomposition. ContraNeRF trains a generalizable radiance field on synthetic 3D-FRONT and tests on real ScanNet. It documents the "sharp but wrong" failure, in which synthetic-trained models place confident density at incorrect depths, and it fights that failure with a contrastive feature loss. That paper motivates our work, but it treats geometry and appearance jointly and has no explicit prior for unobserved volume. G4Splat is the closest method. It extracts planes from sparse views to anchor depth in unseen areas and drives video-diffusion inpainting. Its planes, though, come only from visible evidence in each scene, it optimizes per scene, and it is evaluated in-domain. It does not learn where hidden planes terminate or intersect, and it does not ask what a simulator can teach. PlanarGS and Planar Gaussian Splatting show that planar constraints fix textureless indoor reconstruction, but both work from multi-view observations and cannot synthesize unobserved content. GeoDepth argues that plane-parameter prediction generalizes across domains better than per-pixel depth. That supports our invariance bet for visible surfaces, and we extend it to hidden ones. Two papers constrain the design. Amodal Depth Anything reports that hidden-surface depth is hardest to get right at silhouette edges. The literature on planar failure modes (Freifeld et al., Xie et al.) warns that forcing curved surfaces onto planes produces blocky artifacts. Our design therefore needs a fallback for non-planar regions.

Research Program

The pipeline has three stages.

The first stage lifts the visible content and finds the holes. Visible content is lifted with a monocular or sparse-view depth estimate, and the target view is rendered to obtain a disocclusion mask.

The second stage is the layout predictor. It takes the source image, its visible depth, and the disocclusion mask. It outputs a small set of plane hypotheses, each with a normal $\mathbf{n}$, an offset $d$, and a per-pixel soft assignment over the hidden region. It also outputs a per-pixel "planarity confidence" that routes low-confidence hidden pixels to a non-planar fallback, for which we use an amodal relative-depth model in the style of Amodal Depth Anything. Hidden depth for plane-assigned pixels follows analytically from the plane equation, as in GeoDepth. As a result, the network never regresses free-form depth in the hidden region.

Training data comes from 3D-FRONT and Replica. We render each scene twice, once with furniture and once without, which gives exact hidden background planes, their extents, and their intersection lines. Only geometric inputs and heavily randomized images reach the predictor, and it is supervised only on geometry.

The third stage fills texture. For each revealed plane with parameters $(\mathbf{n}, d)$, we warp the observed part of that plane into a fronto-parallel frame using the plane-induced homography

$$H = K\left(R - \frac{\mathbf{t}\,\mathbf{n}^\top}{d}\right)K^{-1},$$

where $K$ is the camera intrinsics and $(R, \mathbf{t})$ is the rotation and translation to a virtual camera facing the plane. In that frame, perspective foreshortening is removed, and wall or floor texture looks like the stationary, roughly repetitive flat patterns that real-image inpainters handle well. A frozen, real-pretrained diffusion inpainter fills the rectified hole. We then warp the result back and composite it using the predicted layout, so plane intersections stay sharp.

No component that touches appearance is trained on synthetic pixels. The one learned synthetic component sees geometry plus randomized images, and it is evaluated by how well its planes transfer.

Baselines

The naive reference is forward warping of the source view into the target, followed by classical hole filling with the depth-slope trick of the divide-and-conquer DIBR method and Telea inpainting. This shows how much of the problem is solved by anything at all. The only reported number for it is a 2.55 dB PSNR gain over Xu's method on the Flowerpots sequence, which is not comparable to our setting, so we rerun it.

The strongest geometric-generative competitor is RGBD2. It renders an intermediate mesh to novel views and uses an RGB-D diffusion model to fill color and depth in the holes. It is trained on real ScanNet V2 and reports 14.6 dB PSNR at 5% view sparsity on ScanNet V2, against about 10.1 dB for NeRF-based competitors. It is the right opponent because it already solves real-domain disocclusion generatively, with no planar prior and no simulator. We run the released model on our ScanNet test split with the same source views and trajectories.

The strongest sim-to-real competitor is ContraNeRF, trained on 3D-FRONT. It reports 24.81 dB PSNR on ScanNet, against 23.67 dB for IBRNet. That figure is full-image PSNR under its own interpolation-style view protocol, so we rerun it on our wide-baseline splits and report disocclusion-masked metrics.

For the multi-view setting we run G4Splat. It reports Chamfer Distance 6.61 and 23.90 dB PSNR on Replica with 5 views; no reported ScanNet++ number was found on Lacuna for G4Splat. We give it the same views and its default per-scene optimization budget.

All learned baselines receive the same source views, intrinsics, and extrinsics, and the same depth estimator where one is needed. The frozen inpainter we use is also offered to baselines that accept a pluggable inpainter.

Three ablations isolate the key ingredient. The first replaces the learned layout predictor with RANSAC plane fitting on visible depth, extended into the hole. The second inpaints in image space rather than in the rectified frame. The third trains the layout predictor on real ScanNet pseudo-labels instead of synthetic data.

Beating the field means two things. On disoccluded pixels, we must lower hidden-depth error and raise masked LPIPS and PSNR relative to RGBD2 and ContraNeRF. On visible pixels, we must perform no worse.

Why not just…?

Why not just extrapolate the visible planes with RANSAC? This is the most serious threat. It is roughly what G4Splat's planar anchoring does. Extrapolation works when a single wall continues behind a chair. It has no way to decide where a plane stops when two hidden planes meet behind an occluder, such as a wall-floor corner hidden behind a sofa, or when the visible fragment is too small to fit a normal reliably. We will stratify test cases by how many plane intersections fall inside the hole. If learned layout does not beat extrapolation in the multi-intersection stratum, the synthetic component adds nothing, and the honest conclusion is that rectified inpainting alone is the contribution.

Why not just fine-tune an RGB-D diffusion inpainter on real ScanNet, as RGBD2 does? Real reconstructions have holes precisely behind furniture against walls, because those surfaces were never scanned. Real-trained geometry is therefore weakly supervised exactly where disocclusion is hardest. RGBD2 also notes wobbly walls and drift accumulating along trajectories, which analytic plane depth avoids by construction. The third ablation (real pseudo-labels versus synthetic labels versus both) tests this directly. If real pseudo-labels match synthetic ones, the simulator is unnecessary, and we will say so.

Why not just train end-to-end on synthetic with heavy domain randomization or contrastive regularization, as ContraNeRF does? ContraNeRF's own analysis shows that synthetic-trained models learn confident, wrong densities, and that its contrastive loss depends on cross-view feature matches, which break under real motion blur. Our design never asks a synthetic-trained network to produce real appearance. The comparison on real blurry ScanNet frames, split by a blur score, will show whether this separation matters or whether end-to-end regularization closes the gap.

Proposed Experiments

The primary benchmark is wide-baseline single-view and three-view synthesis on ScanNet and ScanNet++. Targets are chosen so that 20–50% of pixels are disoccluded. Ground truth for disoccluded regions comes from held-out frames that observe them, together with the fused mesh, and pixels the mesh never observed are excluded from depth metrics. MuSHRoom's laser scans serve as a second real test with more reliable hidden-surface geometry. Replica provides an in-domain synthetic sanity check.

Metrics are computed separately on disoccluded and visible pixels. They are PSNR, SSIM, and LPIPS; hidden-depth AbsRel; and a planarity residual for pixels labeled wall, floor, or ceiling.

The decisive checks come in four parts. The first is the transfer check: the synthetic-trained layout predictor, measured by hidden-plane normal angle error and offset error on real MuSHRoom and ScanNet, should lose far less accuracy from Replica to real than an equally sized network trained on synthetic data to regress hidden depth directly. The second is the RANSAC-versus-learned comparison in the multi-intersection stratum described above. The third is the rectification ablation, which should improve masked LPIPS mainly on floors and walls seen at grazing angles, where foreshortening is strongest. The fourth is a failure analysis on non-planar hidden content (plants, curtains, curved furniture). There we expect the planarity-confidence gate to route pixels to the amodal-depth fallback. We will report whether the gate prevents the blocky artifacts that uniform planar priors produce, or whether errors there reach the level of the fallback alone.

The proposal is falsified if the synthetic-trained layout offers no gain over RANSAC extrapolation or over real pseudo-label training on real data.