When a 3D object is reconstructed from a single photograph, the unseen sides are usually filled in by a diffusion model, a generator that learns to turn random noise into images step by step. The model is asked to "imagine" extra camera views, and the 3D model is then fit to all of them. These imagined views disagree with each other in structured ways. One view puts a face on the back of a head, which is the Janus problem. When views are produced at high resolution tile by tile, each tile drifts slightly in brightness or depth, and visible seams appear. Current pipelines either average the contradictions away, which blurs the result, or discard whole views that look unreliable, which also throws away their good parts.
Our proposal is to model the disagreement itself. We treat each generated view as the true rendering plus an error image. We then require that error image to be sparse and piecewise constant, meaning it should be zero almost everywhere and made of a few coherent blobs or flat offsets. The tool for this is total variation (TV) minimization, which penalizes the summed magnitude of image gradients and so favors flat regions with sharp borders. A hallucinated face in one view is a compact region of error with a short boundary. A drifted tile is a constant offset whose edges follow the tile grid. Both are exactly the kind of signal TV recovers well. The 3D model is fit only to what remains after the error field has absorbed these contradictions. The TV penalty sits on the error, not on the geometry, so the shape never suffers TV's known tendency to turn smooth ramps into flat steps ("staircasing").
Core Claim
The contradictions in generated multi-view supervision can be rejected region by region, rather than view by view or pixel by pixel, by jointly estimating a 3D representation and one TV-regularized error field per generated view. With $\theta$ the parameters of a 3D Gaussian Splatting model, $y_k$ the $k$-th generated view (RGB, and where available normals and depth), $R_k(\theta)$ its rendering from camera $k$, and $e_k$ the error field for that view, we solve
$$\min_{\theta,\{e_k\}} \sum_k \|y_k - R_k(\theta) - e_k\|_2^2 + \lambda \sum_k \mathrm{TV}(e_k) + \mu \sum_k \|e_k\|_1 .$$
Here $\lambda$ controls how coherent the error regions must be, and $\mu$ controls how much of each view may be declared wrong at all. The observed input view gets no error field; it is the anchor.
Two classical results motivate this form. Convergence Analysis of MAP Based Blur Kernel Estimation shows that squared-gradient priors prefer blurred compromise solutions, while sparse gradient priors make the sharp solution the optimum. Uniformly weighted multi-view fitting is the 3D analogue of the squared-gradient case, and it produces the averaging compromise that ERGO identifies as a driver of floaters, meaning detached clumps of density in empty space. Stable image reconstruction using total variation minimization shows that TV recovery error scales with how well the signal's gradient is approximated by a few terms. That result predicts the method will do well when contradictions are compact, since a blob or a tile offset has few nonzero gradients, and poorly when they are diffuse.
Relation to Prior Work
Four papers supply the setting and the evidence for the gap. ERGO fits 3D Gaussians to multi-view diffusion outputs. It down-weights whole views whose gradient norms suggest irreducible noise, and it adds a visibility-based geometry term and a Sobel-edge texture term. We keep its setting but replace view-level weighting with explicit per-region error maps. A view with a bad face but a good silhouette then still contributes its silhouette.
SPIn-NeRF found that fitting a radiance field to independently inpainted views with squared error fails. Its fix was a perceptual loss (LPIPS) that tolerates misalignment. In-N-Out aligns the diffusion noise across views and uses patch perceptual and adversarial losses. Both treat inconsistency implicitly through a loss that forgives it, whereas we estimate it as a variable the model can inspect. In-N-Out's noise alignment works upstream of our method and is compatible with it.
NeuSD supplies a method we reuse: supervising normal maps separately from color, so that geometry errors are not confused with lighting.
Two papers constrain the scope. Carve3D fine-tunes the multi-view generator itself with a reconstruction-consistency reward. It lowers its consistency score (MRC) from 0.0892 to 0.0606 on the DreamFusion prompts. That result tells us generator-side fixes are strong and orthogonal, so we must show gains on top of a better generator, not only instead of one. DreamBooth3D documents that Janus artifacts can be systematic across rear views. When most views agree on the wrong answer, no consensus-based rejection can help. We treat this as a boundary of the claim and measure it rather than hide it.
For seams, Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models documents tile-level luminance and color drift from independent patch sampling, and shows that overlap averaging blurs the result. Any Resolution Any Geometry removes seams in high-resolution depth and normals with learned cross-tile attention. The research file notes that it has no explicit gradient constraint across tiles and does not build a 3D representation. Our error field gives an explicit, training-free seam model inside 3D fitting.
Research Program
We will build the objective above on a standard single-image-to-3D pipeline. A multi-view diffusion model produces 16 to 32 views around the object. For the high-resolution variant, each view is upsampled with a tiled diffusion refiner, which introduces realistic seams. The error fields live at the resolution of each view, with separate channels for RGB, normals and depth.
Optimization alternates two steps. First, gradient steps on the Gaussians are taken with the error fields held fixed. Second, a proximal update of each $e_k$ is computed with a first-order TV solver of the alternating-projection kind described in Generalized Alternating Projection Based Total Variation Minimization. This update is a 2D problem per view, costs little next to rendering, and is run every few Gaussian iterations.
Early in training the rendering $R_k(\theta)$ is itself poor, so $\mu$ is scheduled from high to low. This prevents the error fields from absorbing real signal before a consensus shape exists.
We will also study a variant in which tile borders are known. The TV weight is reduced along tile edges, which makes a per-tile constant offset nearly free to explain. This tests whether drift and hallucination can be separated inside one field.
The analysis side asks two questions. First, when does consensus identify the correct region? We will measure recovery as a function of how many views carry a contradiction and how large its boundary is, and compare the curves against the prediction from gradient-sparse recovery. Second, do the recovered error maps flag real Janus regions? We will render them and compare them against human annotation of implausible parts on a subset of objects.
Baselines
The naive reference is the same Gaussian Splatting pipeline fit uniformly to all generated views with squared error, plus the usual photometric terms. This is the "averaging compromise" condition and fixes the floor on every metric.
The strongest existing method on this question is ERGO. It works in the same setting, targets the same failure, and reports PSNR 21.37 on Google Scanned Objects. The file gives no SSIM or LPIPS numbers, and no reported result found on Lacuna for ERGO on OmniObject3D in numeric form. We will run ERGO's released procedure if available, and otherwise reimplement its view weighting and its two local objectives. It will use the same multi-view generator outputs, Gaussian initialization, iteration count and hyperparameter search budget as our method, with around twenty tuning runs on a held-out set of GSO objects for each. Beating it means higher PSNR and lower Chamfer distance on GSO at matched wall-clock time, with the PSNR gain larger than the seed-to-seed spread across three generator seeds. We will also report ERGO's published 21.37 alongside our reproduction so that any reproduction gap is visible.
For the seam component, the baselines are overlap-and-average tile blending, as in the patch diffusion work, and per-tile affine alignment of depth and color before fitting. For high-resolution geometry we will also feed URGT-refined normals as supervision. That condition asks whether a learned seam remover upstream makes our error field unnecessary. URGT's own reported UnrealStereo4K result is AbsRel 0.0291, which we cite as context only because that benchmark does not evaluate 3D fits.
Four ablations isolate the key ingredient. The first uses only the sparsity term ($\lambda=0$), which reduces to per-pixel outlier rejection. The second uses only the TV term ($\mu=0$). The third applies TV to the rendered geometry instead of to the error, which is the obvious combination of these directions. The fourth replaces region-level error maps with a single scalar weight per view.
Why not just…?
Why not just use a per-pixel robust loss such as L1, Huber or LPIPS? SPIn-NeRF shows that a perceptual loss rescues NeRF fitting from mildly inconsistent inpaintings, so this is a serious alternative. But a hallucinated face produces moderate, spatially coherent error at every pixel it covers, and no single pixel looks like an outlier to a per-pixel loss. The sparsity-only ablation is exactly this alternative inside our objective. If it matches the full method on the controlled contradiction test described below, the TV term is not earning its place and we will say so.
Why not just down-weight bad views, as ERGO does? View weighting cannot keep the correct half of a half-wrong view. The test is objects where a contradiction covers a minority of a view's pixels. Our prediction is that the per-view scalar ablation and ERGO lose silhouette and texture accuracy there relative to region-level maps. If contradictions in real generator outputs turn out to occupy whole views, view weighting is enough, and the error-map diagnostics will show that.
Why not just fix the generator, as Carve3D does? Generator fine-tuning is a real threat on the Janus side because it removes contradictions at the source. We will rerun our comparison with the most consistent generator available to us. The claim survives only if a gain remains, particularly on tile seams, which a consistency-tuned generator does not address when refinement is tiled.
Why not just put TV on the geometry or density? It is the obvious reading of these four directions. It smooths seams, but it also staircases smooth surfaces, as the TV literature documents, and it does nothing to stop a coherent hallucination from being fit. The TV-on-geometry ablation should show lower normal accuracy on curved GSO surfaces without fewer contradictions.
Proposed Experiments
The first testbed is single-image reconstruction on Google Scanned Objects and OmniObject3D. The metrics are novel-view PSNR, SSIM and LPIPS, Chamfer distance and normal error against the scanned meshes, and the MRC consistency score from Carve3D computed on renders. We add a seam score, defined as the ratio of rendered gradient energy on pixels projected from tile borders to that on interior pixels, which should approach one when seams are gone.
The decisive check is a controlled contradiction test. Into ground-truth GSO renders we paste face-like or foreign-part patches in $m$ of $N$ views, and we add per-tile depth and luminance offsets, with the masks recorded. We then measure the IoU between the recovered error support and the true masks, as well as the final geometry error, as functions of $m/N$ and patch perimeter. The method is supported if mask recovery stays high while contradictions are a minority and degrades with perimeter as gradient-sparse recovery predicts. It should beat the sparsity-only and view-weighting ablations by a clear margin. As $m/N$ passes one half, it should fail, and that failure documents the limit set by systematic Janus errors.
A second testbed uses the 17 BlendedMVS partial-capture scenes from NeuSD with normal-map supervision. This checks that the error field helps surface completion and not only appearance. Finally, a small human study on DreamBooth-style subjects will test whether the recovered error maps coincide with parts that people mark as implausible. That study decides whether the maps are useful as a Janus detector in their own right.
References
- Convergence Analysis of MAP Based Blur Kernel Estimation
- ERGO: Excess-Risk-Guided Optimization for High-Fidelity Monocular 3D Gaussian Splatting
- Stable Image Reconstruction Using Total Variation Minimization
- SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment
- NeuSD: Surface Completion With Multi-View Text-to-Image Diffusion
- Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
- DreamBooth3D: Subject-Driven Text-to-3D Generation
- Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models
- Any Resolution Any Geometry: From Multi-View To Multi-Patch
- Generalized alternating projection based total variation minimization for compressive sensing
- Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items
- BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks