When a 3D scene is reconstructed from photos with Gaussian splatting, a representation built from many small, soft, colored blobs, a common way to make the geometry clean is to compare two estimates of surface orientation. The first is the direction each blob faces. The second is a normal computed from the rendered depth map by differencing neighboring pixels. The second estimate is useful because it lets orientation errors move blob positions, not only rotate the blobs. It breaks at object silhouettes and creases, though. There the finite-difference window straddles two different surfaces and produces a normal that belongs to neither. Current methods handle this by turning supervision down near edges. As a result, the places where geometry matters most for sharp silhouettes and correct reflections receive the least geometric guidance.
We propose a different fix. The depth-to-normal operator itself should decide, for each pixel, which neighbors lie on the same surface. It does this with a soft attention over a small set of candidate neighborhoods, each a directional half-window, and weights each candidate by how well it fits a plane. The resulting normal stays valid right up to a depth step, stays differentiable with respect to depth, and needs no edge downweighting. The same attention weights also give a per-pixel "same-surface" map, which we use to gate the smoothness terms of the training loss so that they stop fighting real discontinuities. Our central claim is testable and narrow: supervising edges with correctly computed normals beats suppressing supervision at edges, both for normal accuracy in a thin band around occluding boundaries and for surface reconstruction metrics.
Where This Sits
The problem is well documented in the file. VCR-GauS shows that normal losses on blob orientations alone do not move blob centers. It introduces "D-Normals" from rendered-depth gradients to couple the two, and downweights the loss when the D-Normal disagrees with a monocular prior. That reweighting is a global disagreement test rather than a treatment of structure. VA-GS adds image-gradient-driven, edge-aware normal consistency that explicitly downweights supervision near edges, inside a multi-term loss with multi-view photometric and feature alignment. UNG-GS learns a per-blob uncertainty scalar that suppresses unreliable normal and depth supervision. All three motivate this proposal: each recognizes that depth-derived normals are wrong at boundaries, and each responds by trusting them less. We instead make them right.
The method draws on two sources. SDA-SNE shows, on precomputed depth maps, that choosing inlier neighbors along multiple directions with a discontinuity penalty cuts angular error by 20–60% near boundaries. Its dynamic programming is discrete, however, and cannot sit inside a differentiable renderer. We keep its insight, that multi-directional neighborhoods should not cross ridges or steps, and replace the hard path search with a softmax over directional plane fits. GeoNet++ supplies the least-squares tangent-plane formulation of depth-to-normal. Its edge handling comes from hand-crafted Canny edges on RGB images, which confuse texture edges with geometry. Our weights come from geometric residuals, not image gradients.
The boundary-attention literature, including reverse attention in SeENet and depth-gradient local attention in MICDrop, operates on 2D feature maps inside networks. We borrow only the principle that aggregation should not cross a boundary. Splatting has no feature backbone into which to place such modules, so we place the attention in the one aggregation step that actually crosses boundaries: the conversion from depth to normals. DisC-GS sharpens silhouettes by clipping blobs with Bézier curves. It is complementary and orthogonal: it improves color at edges but does not produce or supervise normals, and it adds rasterizer complexity that we avoid.
Core Mechanism
For pixel $p$ with rendered depth, we backproject its neighbors into 3D points. We form $K$ candidate supports $S_1,\dots,S_K$ containing $p$: eight half-windows oriented at 45° steps plus the full window, each roughly 5×5. For each support, a weighted least-squares plane gives a normal $\mathbf{n}_k(p)$ and a mean residual $r_k(p)$. The operator's output is
$$\mathbf{n}(p) = \mathrm{normalize}\Big(\sum_{k=1}^{K} a_k(p)\,\mathbf{n}_k(p)\Big), \qquad a_k(p) = \frac{\exp(-r_k(p)/\tau)}{\sum_{j}\exp(-r_j(p)/\tau)},$$
where $\tau$ is a temperature, scaled by pixel depth so that it is invariant to scene scale. On a flat interior every support fits well, so the weights spread and the operator reduces to an ordinary averaged D-Normal. At a depth step or crease, only the supports on $p$'s own side fit a plane, and they take the weight. A single reweighting pass then refits using per-point weights from the selected plane, which handles curved surfaces.
Gradients flow through the plane fits into depth and therefore into blob positions, as in VCR-GauS. The attention entropy $H(p) = -\sum_k a_k \log a_k$ is low where the operator had to choose a side. We convert it to a soft discontinuity map $b(p)$.
The training objective keeps the usual photometric term and adds three geometric terms. The first is a normal-consistency term between blob-orientation normals and attended D-Normals at full weight everywhere, with no edge downweighting. The second is a monocular-prior term with VCR-GauS-style agreement weighting, retained so that the comparison isolates our change. The third is a depth and normal smoothness term multiplied by $1-b(p)$, so that smoothing never pulls across a detected step.
The file identifies "boundary gradient clashing" between isotropic smoothness and edge preservation as an open problem, and the gated smoothness term addresses it directly. Nothing in the objective needs a separate edge detector, and the only new hyperparameter is $\tau$.
Research Program
We first validate the operator alone on ground-truth depth, before any rendering, to establish that it produces correct normals near discontinuities. We then drop it into two existing Gaussian surface-reconstruction codebases, VCR-GauS and VA-GS, replacing only their depth-to-normal computation and their edge-downweighting or smoothing weights. This design means that any gain can be attributed to the operator. Third, we test whether better edge normals translate into better physically based shading. Reflection directions depend on the normal through $\mathbf{r} = \mathbf{v} - 2(\mathbf{v}\cdot\mathbf{n})\mathbf{n}$, so silhouette normal errors should show up as relighting errors. For this we attach the operator to a surfel-based inverse renderer in the style of RadioGS.
Throughout, we report a boundary-band metric alongside standard metrics. Global averages are dominated by flat interiors and can hide exactly the effect under study. The metric is mean angular normal error, and Chamfer distance where meshes allow it, restricted to pixels within a few pixels of ground-truth occluding contours and creases.
Baselines
The naive reference is the plain central-difference D-Normal of VCR-GauS with no edge handling at all. It shows how much the boundary problem actually costs. The strongest existing reconstruction method in the file is VA-GS, which reports average DTU Chamfer distance of 0.49 and Tanks and Temples F1 of 0.54. VCR-GauS reports a Tanks and Temples mean F1 of 0.40. UNG-GS reports DTU Chamfer of 0.62 mm and serves as the uncertainty-based suppression competitor.
We run all methods from their released configurations on the same scenes, image resolutions, and iteration counts. We give our variant no extra tuning beyond a sweep of $\tau$ over five values on two held-out DTU scans, and we give each baseline an equal sweep of its edge-weight hyperparameter. Beating VA-GS means lowering boundary-band normal error substantially while matching or improving its reported 0.49 DTU Chamfer and 0.54 F1. Matching the global numbers while improving the boundary band would still support the claim. Worsening either global number would count against it.
For the standalone operator on the 3F2N depth-to-normal benchmark, the reference is SDA-SNE, which reports roughly 8.1° average angular error on the Hard split against a prior best of about 15.3°. Being differentiable is the point of our operator, so we count it a success if it comes within about a degree of SDA-SNE while being usable inside training. Clearly beating SDA-SNE is not required.
For inverse rendering on the TensoIR synthetic set, the reference is RadioGS at a relight PSNR of 31.41. Normal error on that set has no reported result on Lacuna for RadioGS or TensoIR, so we will measure it ourselves for both.
Three ablations isolate the key ingredient. The first replaces the softmax with a hard argmin over supports. The second removes the entropy gate from the smoothness term. The third keeps our operator but restores VA-GS's edge downweighting. The third ablation tests whether full-strength supervision at edges, rather than the operator alone, drives the gain.
Why not just…?
Why not just use one-sided finite differences, taking the smaller of the left and right gradients? This is the most serious threat. For clean step edges it may recover most of the benefit at almost no cost, so it is run as a first-class baseline. We expect it to fall short in two places. At creases there is no depth jump, so gradient magnitude does not reveal which side to use. Thin structures only a pixel or two wide give neither side a clean support. Our boundary-band metric is split by edge type, steps versus creases versus thin parts, using ground-truth meshes on DTU and the synthetic scenes. If one-sided differences match us on all three types, we will conclude that the attention is unnecessary and report that result.
Why not just keep downweighting at edges, as VA-GS and UNG-GS do? Downweighting removes wrong gradients, but it also removes the right ones, and VCR-GauS's own analysis implies that normal supervision is what moves blob positions. The "our operator plus VA-GS downweighting" ablation settles this directly. If restoring the downweighting costs nothing, the gain comes from the operator's normals alone, and the claim about full edge supervision is wrong.
Why not just run SDA-SNE on the rendered depth and use it as a stop-gradient target? This gives accurate edge normals cheaply. However, the target would then supervise only blob orientations, and VCR-GauS shows that orientation-only supervision does not move centers. We include this variant. The prediction is that it improves boundary-band normal error of the rendered orientations less than our operator improves the Chamfer distance near edges. If it matches us, differentiability through depth is not what matters.
Why not just rely on a stronger monocular normal network? Monocular priors hallucinate and disagree across views, as VCR-GauS documents, and occluding contours are exactly where they are least reliable. Our objective keeps the prior term unchanged in both arms, so any improvement is measured on top of whatever prior is used.
Decisive Checks
The proposal succeeds if three conditions hold. The attended operator must come close to SDA-SNE on 3F2N Hard. Swapped into VCR-GauS and VA-GS, it must cut boundary-band normal error on DTU and the TensoIR synthetic scenes without losing global Chamfer or F1. The gain over one-sided differences must concentrate on creases and thin structures. A secondary check is whether the relighting gap to RadioGS's 31.41 PSNR closes when only the depth-to-normal operator changes, with the error reduction localized along silhouettes in difference images. We will also report training-time overhead against VA-GS's reported 21 minutes. The operator is a fixed local computation with $K$ small plane fits per pixel, so we expect it to add only a small cost to the per-iteration budget. If it adds more than that, we will say so.
References
- VCR-GauS: View Consistent Depth-Normal Regularizer for Gaussian Surface Reconstruction
- VA-GS: Enhancing the Geometric Representation of Gaussian Splatting via View Alignment
- Uncertainty-Aware Normal-Guided Gaussian Splatting for Surface Reconstruction from Sparse Image Sequences
- SDA-SNE: Spatial Discontinuity-Aware Surface Normal Estimation via Multi-Directional Dynamic Programming
- GeoNet++: Iterative Geometric Neural Network with Edge-Aware Refinement for Joint Depth and Surface Normal Estimation
- Towards Bridging Semantic Gap to Improve Semantic Segmentation
- MICDrop: Masking Image and Depth Features via Complementary Dropout for Domain-Adaptive Semantic Segmentation
- DisC-GS: Discontinuity-aware Gaussian Splatting
- Radiometrically Consistent Gaussian Surfels for Inverse Rendering
- TensoIR: Tensorial Inverse Rendering