TL;DR
Neural implicit surface reconstruction typically relies on volume rendering, which integrates features along entire rays and struggles to accurately pinpoint exact surface boundaries. This paper introduces a technique to extract local 3D surface patches during training by "pulling" sampled 3D query points onto the signed distance function's (SDF) zero-level set using gradients. By establishing explicit surface patches, the model can directly enforce multi-view photo consistency, depth alignment, and planar surface-fitting constraints, significantly reducing reconstruction artifacts and geometric drift.
What Problem Does This Paper Address?
Recent state-of-the-art approaches reconstruct 3D geometry from multi-view images by optimizing an implicit neural field (specifically, a Signed Distance Function, or SDF) through volumetric rendering. However, because volume rendering aggregates color and density along entire camera rays via numerical integration, the network never explicitly "senses" the actual 2D surface manifold embedded in 3D space.
As a result, inferred surfaces often drift away from the true geometry—frequently resulting in "inflated" surfaces that float above ground-truth structures—or exhibit phantom artifacts in empty space and smoothed-out details. Existing attempts to supervise surfaces typically check only single ray-surface intersection points, which lack the spatial context required to evaluate local curvature, surface smoothness, or multi-view patch correspondence.
Methods
The authors build on coordinate-based neural SDF models (like VolSDF and MonoSDF) and introduce an explicit surface patch sensing mechanism combined with multi-objective surface supervision:
-
Surface Patch Generation via "Pulling":
Around an anchor point $q$ (either an estimated ray intersection or a back-projected depth cue), the method samples a small cluster of points $p_j \sim \mathcal{N}(q, \tau^2)$. Using the predicted signed distance $d$ and its spatial gradient $\nabla f_\theta$, each point is projected ("pulled") directly onto the zero-level set: $$p'_j = p_j - d \cdot \frac{\nabla f_\theta(p_j)}{||\nabla f_\theta(p_j)||_2}$$ This converts arbitrary 3D queries into an explicit, continuous local surface patch $s = \{p'_j\}$ on the SDF boundary. -
Explicit Surface Constraints:
Once the patch $s$ is established, the network applies three explicit geometric losses: -
Depth Consistency ($L_{DC}$): Projects pulled points $p'_j$ back into the camera view and enforces agreement with sensor-captured or monocular depth priors, discarding points outside the valid camera frustum or depth range.
- Multi-View Photometric Consistency ($L_{NCC}$): Projects the 3D patch into the reference image and neighboring viewpoints, maximizing Normalized Cross-Correlation (NCC) across the top three scoring views to ensure visual consistency across camera angles.
- Confidence-Weighted Surface Fitting ($L_{Fit}$): Enforces local planarity around the anchor point's depth and normal priors. It weights the fit using cosine similarity between the surface gradient and the normal prior ($\eta_j$), which naturally incorporates second-order derivatives to regularize optimization.
- Standard Volumetric Losses:
These patch constraints operate alongside standard volume-rendered RGB ($L_{RGB}$), rendered depth ($L_{DR}$), rendered normals ($L_{Nor}$), and an Eikonal regularization term ($L_{Eik}$).
Ray -> Estimated Intersection -> Gaussian Sampling -> Gradient Pulling -> Surface Patch
│
┌──────────────────────────────────────────────────────────────┤
▼ ▼ ▼
Depth Consistency Photometric Consistency Surface Fitting
(Align with depth) (Multi-view NCC matching) (Local plane alignment)
Datasets
The authors evaluate their method on standard real-world and synthetic indoor benchmarks:
- ScanNet (real-world scans): Evaluated under two standard protocols: the MonoSDF split (supervised via Omnidata monocular depth and normal priors) and the NeuralRGBD split (supervised via sensor-captured depth).
- Replica (synthetic indoor scenes): Evaluated on both test and train splits using monocular cues, as well as compared against dense SLAM systems.
- Tanks and Temples (large-scale indoor scenes): Used for qualitative evaluation against MonoSDF.
No new datasets were introduced.
Key Insights and Ideas
- Bypassing Marching Cubes during optimization: Differentiable pulling allows the network to isolate and manipulate the zero-level set patch locally during gradient descent without running expensive surface extraction algorithms mid-training.
- Overcoming the "Surface Inflation" problem: By enforcing losses explicitly at the zero-level set rather than solely along the integrated ray, the model stops the reconstructed surface from expanding or floating over empty space (a notorious issue in volume rendering).
- Second-order smoothness guidance: Because the pulled points $p'_j$ depend on first-order gradients $\nabla f_\theta$, penalizing normal deviations in the surface fitting loss activates second-order derivatives, leading to significantly smoother planes and better optimization stability.
Key Results
- ScanNet (MonoSDF benchmark): Achieves an F-score of 0.797 and Chamfer-$L_1$ of 0.037, outperforming MonoSDF (0.733 F-score / 0.042 CD), NeuRIS (0.692 F-score), and Neuralangelo (0.292 F-score).
- ScanNet (NeuralRGBD benchmark): Reduces mean Chamfer-$L_1$ error from 11.65 cm (NeuralRGBD) down to 4.60 cm, and increases F-score from 66.90 to 83.79.
- Replica: Outperforms dense RGB and RGB-D SLAM methods (NICE-SLAM, DROID-SLAM, NICER-SLAM), achieving a mean Chamfer-$L_1$ of 2.75 cm and a normal consistency of 93.31%.
- Ablations: Removing the pulling mechanism (applying losses directly to the anchor $q$) caused the F-score to drop from 0.797 to 0.753, confirming that supervising the pulled zero-level set is crucial.
Limitations and Future Work
- Computational Cost with Dense Sampling: Patch query size is bounded (optimal performance was found at $J=9$ points); scaling to larger query counts increases computational and memory overhead without yielding proportional accuracy gains.
- Hyperparameter Sensitivity: Sampling variance $\tau^2$ must be carefully calibrated to the pixel footprint in 3D world space. If set too small, patches fail to capture sufficient local context; if set too large, points fall outside validity masks.
- Dependence on Priors: The pipeline relies on reasonable depth and normal estimates (either monocular priors from networks like Omnidata or sensor inputs) to guide the anchor positions and local plane formulations.
Summary updated