Many methods that rebuild a 3D surface from noisy or incomplete data work like a mosaic. They cut the surface into small patches, fit each patch on its own using a local prior, and then run a global step that stitches the patches into one continuous surface. They then repeat. The global step is almost always a least-squares fit on positions or gradients, and it treats every disagreement between overlapping patches the same way. Perception research says this is the wrong currency. A smooth, low-frequency shift in position is nearly invisible to a viewer. A small high-frequency ripple in the surface normals is glaring under lighting. A missing thin leg is a catastrophe that coordinate distances barely register. Averaging overlapping patches in position space is also "mean-seeking": when two plausible local shapes disagree, it produces their blur rather than either one. Audio research has documented this same failure in over-smoothed synthesis.
We propose to change only the global step of a patch-based reconstructor. It will reconcile patches in shading space rather than position space, and it will select among competing local patch hypotheses rather than average them. Every patch is shaded under a few fixed lights in its own local coordinate frame, so no multi-view rendering is needed and no region is hidden from the loss. The same design gives per-patch diagnostics: a record of where the stitched surface looks different from what the local prior believed. A second contribution is measurement. We track geometric metrics, learned perceptual metrics, and human judgments across the outer iterations of the alternating loop. This tests whether the "objective metric converges early, perception keeps improving" divergence reported in audio also occurs in surface reconstruction.
Core Claim
In local-global mosaic reconstruction, the perceptual damage happens mostly in the global stitching step, not in the local fit. Two properties of that step are responsible. It measures disagreement in coordinates, and it resolves disagreement by averaging. Replacing it with a stitching energy has two consequences. First, the surface is free to drift in low-frequency position but must reproduce each selected local patch's appearance under basis lighting. Second, overlapping regions are resolved by discrete choice among local proposals. We expect this to preserve thin structures and fine relief that the standard pipeline erases, at comparable Hausdorff error. We also expect it to be measurably preferred by human viewers.
Where This Sits
The mosaic side supplies the machinery we reuse. Meshlet Priors for 3D Mesh Reconstruction learns a variational autoencoder over about 2.2 million small canonical surface patches ("meshlets"). It alternates between fitting meshlet codes to noisy points and aligning them into one watertight mesh, and reports a Hausdorff error of 0.054 against 0.293 for AtlasNet and 0.304 for OccNet. Its local search and its global alignment both use spatial distance, and the authors name optimization cost and loss of fine and thin detail as limitations. We keep its local step and prior unchanged and replace the global step.
From Shading to Local Shape showed that local patches admit a distribution of shape proposals that global reasoning must choose among. That is the precedent for our selection-based stitching. The surface-from-gradients line, Surface-from-Gradients: An Approach Based on Discrete Geometry Processing, integrates per-facet normals with a single uniform least-squares solve. Our global step also works in the normal and shading domain, but it is robust and selective rather than uniform and averaging.
The perception side motivates the design and constrains it. Perceptual Deep Depth Super-Resolution shows that depth error and perceived quality can point in opposite directions: a 46 mm RMSE result looks bumpy while a 99 mm result looks right. It fixes this by comparing normals shaded under three orthogonal Lambertian lights. We borrow exactly that shading comparison. That paper applies it globally to single 2.5D depth maps; we apply it per patch, in each patch's canonical frame, inside a stitching solve for full closed surfaces.
DR-KFS: A Differentiable Visual Similarity Metric for 3D Shape Reconstruction shows that Chamfer distance melts thin structures and that a keypoint-matching loss over 25 rendered views fixes much of this, with 71% human preference overall. It is a whole-object loss for feed-forward networks. It pays for 25 renders per step, cannot see surfaces hidden from every camera, and does not say which patch is at fault.
The learned evaluators Textured Geometry Evaluation: Perceptual 3D Textured Shape Metric via 3D Latent-Geometry Network (TGE) and SRAM: Shape-Realism Alignment Metric for No Reference 3D Shape Evaluation output one score per object. We use them only as independent evaluation signals, never as training losses, to avoid optimizing the judge.
The audio side provides the lens, not the method. Geometric Iterative Retrieval for Neural Audio Codec Resynthesis reports two things. Log-spectral distance plateaus after 3 of 9 refinement layers while listeners prefer all 9. One-step regression is mean-seeking and muffles detail. Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction shows that fixing only diagnosed local segments beats full re-generation 77% of the time. We take two concrete hypotheses from these papers. Averaging in the global step is the geometric analogue of mean-seeking. Iteration-count choices made by watching geometric error may stop too early, or too late, for perception. We do not import an LLM judge: the file documents that such judges are expensive, and our per-patch shading residual is a cheap, local diagnostic instead.
Research Program
The local step is unchanged from the meshlet method, with one difference: we keep the top $K$ latent codes per patch rather than one. These come from multiple restarts and nearby samples in the prior, giving each patch a small set of plausible local shapes.
The global step then solves for a surface $S$ and a choice $c_i \in \{1,\dots,K\}$ for each patch $i$ by minimizing
$$E(S,c)=\sum_i \sum_{k=1}^{3} \big\| \rho(n_S,\ell_k^{(i)}) - \rho(n_{i,c_i},\ell_k^{(i)}) \big\|_{1,P_i} \;+\; \lambda \big\| G * (x_S - x_c) \big\|_2^2 .$$
Here $P_i$ is the footprint of patch $i$ on the surface, and $n_S$ and $n_{i,c_i}$ are the normals of the stitched surface and of the chosen proposal. The function $\rho(n,\ell)=\max(0,n\cdot\ell)$ is Lambertian shading under light $\ell$. The lights $\ell_k^{(i)}$ are three orthogonal lights fixed in patch $i$'s canonical frame. The term $x_S - x_c$ is the positional offset between the surface and the chosen proposals, $G$ is a low-pass filter, and $\lambda$ weights it.
Each term does a specific job. The first term demands that the surface look locally like the selected proposal. Because it uses an L1 norm, a region where proposals conflict is pulled toward one of them rather than their average. The second term ties the surface to the data only in low-frequency position, where the perception literature says deviations are cheap. We solve by alternating two moves. Discrete label updates for $c$ use graph-cut-style moves over the patch adjacency graph. A continuous surface update then solves a reweighted least-squares problem over vertices, reusing the meshlet method's watertight extraction.
The per-patch value of the first term after stitching is a spatial defect map. It marks patches where the global surface departs visibly from local belief. In the next outer iteration we re-fit only the top fraction of flagged patches, which is the "diagnose, then refine" idea implemented with a local, rendering-free signal.
We will release the modified reconstructor, per-iteration mesh snapshots, and the human judgments.
Baselines
The naive reference point is the unmodified Meshlet Priors for 3D Mesh Reconstruction pipeline. On its benchmark of 20 decimated, noise-corrupted objects it reports Hausdorff 0.054. It is the strongest competitor on the geometric axis because our method shares its local step and prior, so any difference is attributable to stitching. We will rerun it from the released configuration with the same number of outer iterations and wall-clock budget as ours. If our rerun does not approximately reproduce 0.054, we report both numbers. Beating it means human preference significantly above 50% in two-alternative forced choice, together with Hausdorff no worse than its value plus a pre-registered tolerance of 10% relative.
The strongest existing perceptual method is DR-KFS, applied as an added whole-mesh loss during the meshlet global step, using its 25 rendered views. It is the most direct competitor because it already fixes the thin-structure failure of coordinate losses. We give it an equal wall-clock budget, and a matched-quality run with unlimited budget to expose the cost difference. Its reported figure is 71% human preference over Chamfer-trained reconstructions in feed-forward networks. No reported result was found on Lacuna for DR-KFS inside a meshlet or other mosaic pipeline. Beating it means preference at least at parity with lower compute, or better preference at equal compute.
A third baseline adds the three-light shading loss of Perceptual Deep Depth Super-Resolution to the existing positional global step, rendered globally from a fixed camera set, with no selection. Two ablations isolate our ingredients. The first uses shading-space stitching with $K=1$, meaning no selection: it separates "change the currency" from "stop averaging." The second uses $K>1$ selection under positional rather than shading energy. A last ablation replaces defect-map-targeted refitting with uniform refitting of all patches at equal compute.
Why not just…?
Why not just tune the smoothness weight or run more iterations of the original meshlet loop? The meshlet paper itself says hand-set smoothness trades noise removal against detail. Our divergence study directly measures whether more iterations help perception. This alternative is a real threat. If a swept regularization weight plus more iterations matches our human preference, the conclusion is that iteration scheduling, not a new energy, is the fix. We sweep both on held-out objects at equal compute and report the full frontier.
Why not just add DR-KFS or a global shading loss to the existing pipeline? This may recover much of the gain. Its known weaknesses are cost, blindness to surfaces hidden from all cameras, and the absence of per-patch credit, as noted in DR-KFS. None of these proves it loses on preference. Baselines two and three settle it. We additionally report preference restricted to self-occluded regions such as chair undersides and lamp interiors, where our canonical-frame shading should be the difference.
Why not just post-process the meshlet output with normal-based detail enhancement? Methods such as 3D Surface Detail Enhancement from a Single Normal Map can sharpen relief after the fact. They cannot restore a thin leg or edge that averaging has already removed, because the information is gone from the output. We test this by applying post-hoc enhancement to the baseline and comparing thin-structure recall against the reference mesh, measured as the fraction of reference parts under a width threshold that survive.
Why not just optimize TGE or SRAM directly? Both output one score per object, which gives no spatial credit and invites overfitting to the evaluator. Keeping them as held-out judges is what makes our evaluation meaningful.
Proposed Experiments
The first experiment is a premise check. On the Colored Shape Fidelity dataset, we aggregate the patch-wise shading residual against the reference mesh into one full-reference score and correlate it with human judgments. We compare against the reported PLCC of 0.6355 for Chamfer distance and 0.7887 for TGE from Textured Geometry Evaluation. Our score ignores texture on a textured benchmark, so we do not expect to reach TGE. But if it does not clearly exceed Chamfer distance, the premise that shading-space disagreement tracks perceived geometric quality fails, and we stop.
The second experiment is reconstruction. We use the meshlet benchmark objects and ShapeNet chairs, lamps and airplanes chosen for thin parts, under several noise and sparsity levels. We report Hausdorff and Chamfer distance, the three-light shading error, thin-structure recall, TGE computed with uniform color, and SRAM realism, whose reported PLCC is 0.69 on RealismGrading. The primary endpoint is a pairwise human study on turntable renderings under novel lighting, with lighting and viewpoints disjoint from the three canonical lights used in optimization so the loss cannot be gamed by the display. The comparisons are ours against each baseline, with preregistered sample sizes.
The third experiment tests the divergence. We save meshes at every outer iteration of both the baseline and our method, collect human rankings across iterations for a subset of objects, and plot them against Hausdorff and TGE curves. The GIR-style hypothesis predicts that geometric error plateaus while human-ranked quality keeps changing. A finding that they plateau together is also informative: it would say the misalignment in 3D lives in the loss, not the stopping rule.
The decisive outcomes are these. If shading-space stitching with $K=1$ captures most of the gain, the story is about the metric used for stitching. If selection with $K>1$ is needed, the story is about mean-seeking. If the tuned-smoothness sweep matches both, the proposal's central claim is wrong, and we will say so.
References
- Meshlet Priors for 3D Mesh Reconstruction
- From Shading to Local Shape
- Surface-from-Gradients: An Approach Based on Discrete Geometry Processing
- Perceptual Deep Depth Super-Resolution
- DR-KFS: A Differentiable Visual Similarity Metric for 3D Shape Reconstruction
- Textured Geometry Evaluation: Perceptual 3D Textured Shape Metric via 3D Latent-Geometry Network
- SRAM: Shape-Realism Alignment Metric for No Reference 3D Shape Evaluation
- Geometric Iterative Retrieval for Neural Audio Codec Resynthesis
- Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
- 3D Surface Detail Enhancement from a Single Normal Map