Abstract
It is vital to recover 3D geometry from multi-view RGB images in many 3D computer vision tasks. The latest methods infer the geometry represented as a signed distance field by minimizing the rendering error on the field through volume rendering. However, it is still challenging to explicitly impose constraints on surfaces for inferring more geometry details due to the limited ability of sensing surfaces in volume rendering. To resolve this problem, we introduce a method to infer signed distance functions (SDFs) with a better sense of surfaces through volume rendering. Using the gradients and signed distances, we establish a small surface patch centered at the estimated intersection along a ray by pulling points randomly sampled nearby. Hence, we are able to explicitly impose surface constraints on the sensed surface patch, such as multi-view photo consistency and supervision from depth or normal priors, through volume rendering. We evaluate our method by numerical and visual comparisons on scene benchmarks. Our superiority over the latest methods justifies our effectiveness.
Introduction
Recovering high-quality 3D geometry from a set of 2D images is a cornerstone of computer vision. Recent "neural implicit" methods, such as NeRF, NeuS, or VolSDF, represent geometry as a Signed Distance Function (SDF) and optimize it using volume rendering. However, volume rendering often suffers from a "surface awareness" problem. Because it integrates color and density along an entire ray to predict a pixel's color, the optimization doesn't always precisely locate the surface interface. This lack of explicit surface sensing often leads to surface drift, artifacts like "floating" geometry in empty space, and a general lack of fine-grained geometric detail.
Methodology
To address these limitations, we propose a method to "sense" a local surface patch rather than just a single point along a ray. The core of our approach is a differentiable "pulling" mechanism and the application of explicit surface constraints.
The Pulling Mechanism
For any given ray, the model estimates the intersection point with the surface. It then samples a cluster of points around this anchor. Using the current SDF's predicted distances and gradients, the model "pulls" these points onto the zero level set—the mathematical surface. This process creates a local 3D patch $s$ that represents the geometry more holistically than isolated points.
Explicit Surface Constraints
Once the patch is established, we apply three specific types of supervision directly to it:
- Depth Consistency: Ensures the points on the patch project back to the correct depths in the input views.
- Photometric Consistency: Uses Normalized Cross Correlation (NCC) to ensure the patch looks consistent when viewed from different camera angles.
- Surface Fitting: Encourages the patch to be locally smooth and aligned with normal priors, utilizing second-order derivatives to find better geometric solutions.
This hybrid optimization combines these new surface-level constraints with standard volume rendering losses (RGB, Eikonal, etc.), creating a more robust training signal.
Evaluation
We evaluated our approach on three major benchmarks for indoor and large-scale scene reconstruction: ScanNet, Replica, and Tanks and Temples.
Key Results
The proposed method, Surface-Sensing-SDF, achieved state-of-the-art (SOTA) performance across several metrics. On ScanNet, it achieved an F-score of 0.797, significantly surpassing MonoSDF's 0.733. Qualitative results show much "cleaner" reconstructions; while previous methods often produced "inflated" or floating surfaces, our method produces compact, tight-fitting geometry with high visual fidelity.
Conclusion
By transitioning from supervising individual points along a ray to supervising local surface patches, our method captures local curvature and texture consistency that point-based methods miss. Leveraging the SDF gradient to "snap" query points onto the geometry allows for differentiable surface sensing that significantly improves the accuracy and completeness of 3D reconstructions. Future work may explore extending this patch sensing logic to more complex material representations or integrating it with faster rendering techniques like 3D Gaussian Splatting.
Summary updated