Repainting Ground Truth: Measuring and Fixing the Trade-off Between Blank Walls and Thin Wires

3D reconstruction from photographs fails in two well-known ways. Blank surfaces, such as white walls and uniform floors, give the cameras nothing to match, so the recovered geometry warps or fills with fog. Thin objects, such as cables, chair legs and railings, cover so few pixels that the optimizer treats them as noise and smooths them away. The usual fix for blank surfaces is to add a prior: a predicted depth or surface direction from a single-image network, or an assumption that the surface is flat. That prior pulls geometry toward something smooth. We think this fix and the second failure are the same problem seen from two sides. A prior strong enough to flatten a blank wall is also strong enough to erase the cable hanging in front of it. Current benchmarks cannot measure this trade-off, because in real scenes texture and geometry vary together. A wall is blank because it is a wall, and a cable is hard because it is both thin and dark against a pale background.

The proposal has two parts. The first is a measurement tool. We take scenes whose true geometry is known, and we use depth-conditioned texture synthesis to repaint each part of the scene with a controlled amount of texture. Depth-conditioned texture synthesis means a diffusion model (an image generator that learns by gradually removing noise) that is guided by depth maps of a fixed mesh. Because the texture is baked onto the true mesh before rendering, every view agrees with every other view by construction. This lets us vary "how textured is the wall" and "how much does the wire contrast with the wall" independently, while the geometry stays fixed. The second part is a method. The diagnosis it targets is that existing methods decide where to trust priors using clues from a single image. Our method instead decides using whether the multiple views actually pin down depth at that pixel.

Core Claim

The claim has two halves. First, among current regularized reconstructors, thin-structure recall and accuracy on blank surfaces trade off monotonically against prior strength. This trade-off only becomes visible when texture is controlled separately from geometry. Second, the single-image clues used to switch priors off cannot resolve the trade-off. DN-Splatter, for example, relaxes its depth prior wherever the image has a strong color edge. That rule fails in two predictable cases. Painted patterns on a flat wall have edges, so the useful prior is wrongly switched off there. A low-contrast thin structure may have no strong edge, so the prior is left on and erases it. A gate based on how sharply the multi-view matching cost peaks in depth avoids both errors.

Grounding in prior work

Three papers motivate the blank-surface side, and all three regularize globally. MonoSDF supervises a neural signed distance field (a network that outputs distance to the nearest surface) with monocular depth and normals from Omnidata. The research file notes that this smooths away cables and slender legs. DN-Splatter adds depth and normal priors to Gaussian splatting (a scene made of many small, fuzzy colored blobs) and relaxes the depth prior near image edges. It is the closest existing gating idea, and we test it directly. 3D Gaussian Flats snaps blobs onto detected planes, which the file notes can flatten thin structures near walls. StructNeRF enforces planarity within superpixels and shares this weakness.

On the thin side, Geo-Neus recovers thin handles by supervising the surface directly with sparse 3D points and patch matching across views. Mesh Splatting recovers masts and foliage with a softened mesh. Both depend entirely on photometric evidence, meaning color differences between views, and both get no useful signal on blank expanses. These papers constrain our design: on surfaces that are truly blank, photometry cannot be the only signal, so a prior must remain.

On the synthesis side, Text2Tex and TextMesh supply the method we reuse. Both paint diffusion-generated texture onto a clean, fixed mesh, guided by its depth maps. We repurpose this as a controlled data generator rather than as an asset-creation tool. Their stated weakness, lighting baked into the texture, is harmless here because we re-render under known lighting.

GHOST is the obvious competing link between texture synthesis and blank-surface reconstruction. It invents opaque texture on transparent objects, one frame at a time, so that off-the-shelf models work. We deliberately do not follow that route for opaque blank walls. On a truly uniform wall, texture invented from a single image carries no geometric information beyond what a monocular depth network already provides, and invented textures disagree across frames. The section "Why not just…?" below develops this point.

Research Program

The benchmark, which we call RetexScenes, starts from scenes with dense ground-truth geometry: ScanNet++ laser meshes for indoor scenes and DTU scans for objects. We segment each mesh into parts using its semantic labels. We tag thin parts automatically: a part counts as thin where its local thickness, measured from the medial axis (the skeleton running through the middle of the shape), falls below a few voxel widths at the reconstruction resolution.

For each part, we assign one of several texture conditions, set independently per part. The conditions are flat albedo (no texture), low-frequency gradients, natural texture generated by Text2Tex-style depth-guided diffusion, and high-frequency procedural noise. Separately, we set a contrast level between each thin part and the background behind it. We then render the camera trajectories of the original captures with a physically based renderer, including shading. The result is a grid of scenes in which only appearance changes. Two sources of evidence remain even on flat-albedo parts, and we treat both as part of the measurement rather than eliminating them. Shading still varies across a blank surface, and thin structures still occlude the background differently from different viewpoints.

The method, evidence-gated priors, works as follows. At each training pixel, take the currently rendered depth $d$ and evaluate the multi-view normalized cross-correlation cost $C$ (a patch-matching cost across views) at depths $d-\delta$, $d$ and $d+\delta$. Normalized cross-correlation compares the same image patch across neighboring views. The sharpness of the cost minimum is the curvature

$$\kappa = \frac{C(d+\delta) - 2C(d) + C(d-\delta)}{\delta^2}.$$

A flat cost curve means the views do not determine depth at this pixel, and a sharp minimum means they do. The prior's weight is $w = \exp(-\kappa/\tau)$, where $\tau$ is a temperature that sets how quickly the prior is switched off. The training loss is the usual photometric loss plus $w$ times the depth-and-normal prior term. We use the same $w$ to stop the pruning and opacity-decay steps from deleting blobs where $\kappa$ is high.

We implement this in two settings: as a modification of DN-Splatter for Gaussian splatting, and of MonoSDF for neural signed distance fields. This tests whether the gate helps independently of the scene representation. The extra cost is two additional cost evaluations per sampled pixel. That is far cheaper than the roughly 10 hours per scene reported for Geo-Neus.

Baselines

The naive reference point is vanilla 3D Gaussian splatting with no priors. DN-Splatter reports that it reaches an F-score of 0.5835 on MuSHRoom, which shows how badly unregularized methods do indoors. This baseline should recover thin structures relatively well on high-contrast settings and fail on blank parts, which anchors one end of the trade-off.

The strongest competitor on the actual question is DN-Splatter, because its edge-aware depth loss is already a heuristic for deciding where priors apply. It reports an F-score of 0.9243 on MuSHRoom, against 0.6378 for SuGaR.

We also run three other strong methods. MonoSDF reports a Chamfer distance of 0.042 on ScanNet. 3D Gaussian Flats reports that depth RMSE drops from 0.44 to 0.27 on ScanNet++. Mesh Splatting is a prior-free thin-structure specialist; no reported result was found on Lacuna for it on DTU Chamfer distance, so we use our own runs of its released settings.

The ablation that isolates our key ingredient keeps everything in our DN-Splatter variant fixed and swaps only the gate. We compare four gates: our multi-view curvature gate, DN-Splatter's RGB-gradient gate, a constant prior weight swept over a range of values, and an oracle gate. The oracle uses ground-truth knowledge of which parts are blank, which bounds how much any gate could gain.

For fairness, all methods receive the same images and poses, the same Omnidata priors, and the same iteration budget on one GPU type. Each method gets an equal hyperparameter search of about 10 configurations, tuned on held-out RetexScenes scenes and never on the test scenes.

Beating the baselines means the following. On the full grid of texture conditions, our curve of thin-part F-score against blank-part depth error must lie strictly beyond, or Pareto-dominate, the curve traced by sweeping DN-Splatter's prior weight. On real MuSHRoom, ScanNet++ and DTU, we must match or improve the reported F-score or depth RMSE while improving thin-part recall.

Why not just…?

Why not just use real benchmarks with part masks? Real scenes tie texture to geometry, so a gain on thin parts cannot be separated from the fact that those parts happened to be high-contrast. The ETH3D benchmark also documents smeared and incomplete reconstruction of cables even with 24-megapixel inputs. This makes real-scene thin-part ground truth the weakest exactly where we need it most. We still report real-data results, but the synthetic grid is what supports causal statements.

Why not use procedural noise or flat colors instead of diffusion textures? This is a real threat to the synthesis component. If method rankings on procedural textures match rankings on diffusion textures, then diffusion is unnecessary and we will say so. Our hypothesis is that natural textures differ from noise in ways that matter for matching. They contain repeated motifs that produce false matches, and they have soft gradients that resemble shading. Both effects should make edge-based gates misfire in ways that noise does not. The procedural condition is included in the grid specifically to test this.

Why not hallucinate texture, as GHOST does, so the views can be matched? On an opaque, uniform wall, an image-space hallucinator conditioned on that same image can only encode monocular shape cues. MonoSDF already supplies those directly. Its per-frame outputs are also not consistent across views. GHOST's own gains come from transparent objects, where the input is misleading rather than empty. Painting texture onto the current estimated geometry instead is circular, because the rendered views agree with whatever geometry was painted. As a control, we include a GHOST-style per-view texture hallucination on our blank-wall conditions.

Why not just tune DN-Splatter's edge threshold or prior weight? The prior-weight sweep in our baselines is exactly this experiment. Our prediction is that no single setting reaches the frontier that our gate reaches, because the single-image edge signal is wrong in the two specific cases named in the core claim. If tuning closes the gap, the method claim fails and the benchmark finding stands on its own.

Proposed Experiments

The first decisive check tests the trade-off itself. We sweep prior strength for MonoSDF, DN-Splatter and 3D Gaussian Flats across RetexScenes. We measure F-score restricted to ground-truth points on thin parts and depth RMSE restricted to flat-albedo parts. We then plot one against the other. A monotone frontier confirms the coupling. Its absence would mean the two failures are independent, which would undercut the premise of the proposal.

The second check tests the gate. We compare the four gates on the cells of the grid that discriminate between them: patterned flat walls, and low-contrast thin parts in front of blank walls. In these cells, the RGB-gradient gate should lose to ours, and ours should approach the oracle.

The third check tests whether the synthetic findings transfer to real data. We rank all methods on RetexScenes and on real MuSHRoom, ScanNet++ and DTU. We then compute the rank correlation between the two orderings, alongside the reported real-data numbers listed above. If the gate improves synthetic thin-part recall but not real recall, we will report the benchmark as the contribution and treat the method as unproven.