The Limitations of Isolated Estimation
When reconstructing 3D scenes from 2D images, computer vision models traditionally treat depth estimation (predicting the distance of each pixel from the camera) and surface normal estimation (predicting the 3D orientation vector of each surface) as separate, independent tasks.
However, solving these problems in isolation introduces systematic geometric artifacts. As noted by Michael Strecke et al. in Accurate Depth and Normal Maps from Occlusion-Aware Focal Stack Symmetry, independent depth networks often suffer from "piecewise flat" distortions because they lack an explicit understanding of surface orientation. Conversely, attempting to derive surface normals post-hoc by calculating the spatial gradients of a predicted depth map severely amplifies high-frequency noise and quantization errors.
An alternative historical approach has been sequential workflows, where surface normals are estimated first and subsequently integrated to compute a depth map. However, as highlighted by Roy Or-El et al. in Real-Time Depth Refinement for Specular Objects, these two-step pipelines are computationally intensive and prone to accumulating errors during the integration phase, leading to global geometric distortions and hallucinated geometry.
The Mathematical Coupling of Geometry
The core insight behind joint estimation is that depth and surface normals are fundamentally coupled by the laws of differential geometry: a surface normal is mathematically related to the spatial gradients of the underlying depth field. By optimizing both properties simultaneously, models can enforce strict physical consistency.
To bind these variables together, researchers employ explicit geometric constraints. For instance, in SURGE: Surface Regularized Geometry Estimation from a Single Image, Peng Wang et al. enforced a mathematical orthogonality constraint: the vector connecting any two points on a predicted 3D plane must be strictly orthogonal to the surface normal at those points. This mutual constraint ensures that the spatial distribution of predicted depth points directly conforms to the predicted surface orientation.
Similarly, Mattia Rossi et al. in Joint Graph-Based Depth Refinement and Normal Estimation incorporated the slope of the depth map directly into a variational regularization framework. By aligning depth gradients with predicted normals, the joint optimization framework allows high-frequency details to sharpen depth boundaries. This mathematical alignment enables models to recover intricate local structures—such as clothing folds in the work by Yasamin Jafarian et al. (Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos) or fine object boundaries from raw sensor data as shown by Rongrong Gao et al. (Joint Depth and Normal Estimation from Real-world Time-of-flight Raw Data).
Complementary Strengths: Global Layout vs. Local Detail
Joint estimation works remarkably well because depth maps and surface normals possess highly complementary strengths that compensate for each other's failure modes:
- Depth maps excel at capturing coarse, global spatial layouts and establishing metric scale, but they tend to smooth out fine structural details and struggle in textureless regions.
- Surface normals are highly sensitive to local orientation changes and high-frequency surface textures, making them excellent at capturing sharp edges, but they lack global context and absolute depth information.
When combined, these cues resolve fundamental ambiguities. In MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction, Zehao Yu et al. demonstrated that using monocular depth to constrain global structure alongside normal maps to recover fine-grained details solves the "white wall problem"—large, textureless regions where standard color-based reconstruction fails. Likewise, Dongxu Zhao et al. in MVPSNet: Fast Generalizable Multi-view Photometric Stereo utilized a multi-task approach to extract both properties from a shared feature volume, capturing global geometry through depth volumes while preserving sharp local features through normal estimation.
Architectural Strategies for Mutual Supervision
Modern deep learning architectures exploit this synergy by designing networks that allow depth and normal representations to interact directly during training and inference.
Rather than relying on noisy analytical derivations, Huangying Zhan et al. in Self-supervised Learning for Single View Depth and Surface Normal Estimation trained dedicated parallel networks for depth and normals using a joint geometric consistency loss. This strategy ensures that both networks provide mutual supervisory signals, forcing the predicted geometry to adhere to physical laws.
To minimize computational overhead, models can also perform this joint estimation within a shared latent space. As shown by Jiyuan Wang et al. in From Editor to Dense Geometry Estimator, leveraging global attention mechanisms over a shared representation allows the network to learn highly consistent structural priors. Finally, Zhiheng Fu in Multi-stage information diffusion for joint depth and surface normal estimation demonstrated that forcing these tasks to interact late in the inference process guarantees that the final surface orientations are physically plausible and mathematically aligned with the estimated distances.
Key Foundational Work
Accurate Depth and Normal Maps from Occlusion-Aware Focal Stack Symmetry (2017) — contributed 2 concept elements to this direction.
SURGE: Surface Regularized Geometry Estimation from a Single Image (2016) — contributed 2 concept elements to this direction.
Real-Time Depth Refinement for Specular Objects (2016) — contributed 1 concept elements to this direction.