Abstract
In this paper, we propose a geometric neural network with edge-aware refinement (GeoNet++) to jointly predict both depth and surface normal maps from a single image. Building on top of two-stream CNNs, GeoNet++ captures the geometric relationships between depth and surface normals with the proposed depth-to-normal and normal-to-depth modules. In particular, the "depth-to-normal" module exploits the least square solution of estimating surface normals from depth to improve their quality, while the "normal-to-depth" module refines the depth map based on the constraints on surface normals through kernel regression. Boundary information is exploited via an edge-aware refinement module. GeoNet++ effectively predicts depth and surface normals with high 3D consistency and sharp boundaries resulting in better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve 3D reconstruction quality and pixel-wise accuracy of depth and surface normals. Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth prediction in 3D. In contrast to current metrics that focus on evaluating pixel-wise error/accuracy, 3DGM measures whether the predicted depth can reconstruct high quality 3D surface normals. Our experiments on NYUD-V2 and KITTI datasets verify that GeoNet++ produces fine boundary details and the predicted depth can be used to reconstruct high quality 3D surfaces.
Introduction
Estimating the 3D structure of a scene from a single 2D image is a fundamental challenge in computer vision. Traditionally, researchers have treated depth estimation and surface normal estimation as independent tasks. However, this "siloed" approach often results in geometric inconsistency. For instance, a depth map might contain small artifacts on a flat surface while the corresponding predicted surface normals are perfectly planar. This lack of agreement leads to poor 3D reconstructions, characterized by "flying pixels" at object boundaries or distorted planar surfaces. Furthermore, standard evaluation metrics like RMSE focus on pixel-wise accuracy but fail to capture how well the predicted depth represents the underlying 3D geometry.
Methodology
GeoNet++ addresses these issues by enforcing the mathematical relationship between depth and normals through a two-stream CNN architecture and several novel refinement modules.
Geometric Consistency Modules
- Depth-to-Normal Module: This module treats the estimation of normals from depth as a least-squares problem. It calculates the tangent plane for a local neighborhood of 3D points derived from the depth map. By baking this mathematical constraint directly into the network, the model learns to produce normals that are geometrically consistent with the depth.
- Normal-to-Depth Module: Conversely, this module uses the predicted surface normals to refine the depth map. It employs kernel regression, where neighboring pixels contribute to the depth of a central pixel based on the local surface orientation.
Refinement and Iteration
To resolve blurry object boundaries, an Edge-Aware Refinement module uses a Canny edge detector to guide a recursive propagator, sharpening edges and smoothing noise in flat regions. The network can be applied iteratively, using its own refined outputs as the next stage's inputs to progressively improve 3D consistency.
3D Geometric Metric (3DGM)
The authors introduce the 3D Geometric Metric (3DGM) to better evaluate reconstruction quality. Unlike 2D metrics that look at raw depth values, 3DGM evaluates how well the predicted depth can be used to reconstruct high-quality 3D surface normals, providing a more natural proxy for real-world 3D application performance.
Experimental Results
The model was evaluated on the NYUD-V2 (indoor) and KITTI (outdoor) datasets. GeoNet++ achieved state-of-the-art results, reducing the mean angle error for surface normals on NYUD-V2 to 18.5° and achieving an RMSE of 4.10 on KITTI.
Visually, the results are striking. Where previous methods like DORN produced noisy 3D point clouds, GeoNet++ produces clean, flat planes for walls and floors and sharp, distinct edges for furniture and poles. The surface normals act as a regularizer for the depth, and vice versa, improving the accuracy of both tasks simultaneously.
Conclusion
GeoNet++ demonstrates that explicitly modeling geometric priors within a neural network significantly improves the consistency and quality of 3D scene reconstruction. While iterative inference adds some computational overhead, the framework is generic and can be integrated into broader pipelines for stereo matching, SLAM, or robotics where geometric fidelity is paramount.
Summary updated