In computer vision, reconstructing a 3D scene from 2D images often relies on two primary geometric signals: depth (the distance of a point from the camera) and surface normals (the orientation of the surface at that point). Traditionally, these were treated as separate tasks or estimated sequentially. However, a growing body of research demonstrates that depth and normals are two sides of the same coin, and estimating them jointly is the key to recovering high-fidelity 3D geometry.
The Complementary Nature of Global and Local Cues
The fundamental intuition behind joint estimation is that depth and normals provide complementary information. Depth maps are excellent at capturing the global structure and spatial layout of a scene. However, as Zehao Yu and colleagues point out in MonoSDF, depth estimation often struggles with the "white wall problem"—large, textureless regions where RGB signals are ambiguous.
Surface normals, by contrast, provide high-frequency local information. While they don't tell you exactly how far away an object is, they describe the fine-grained curvature and orientation of its surface. Yasamin Jafarian and Hyun Soo Park demonstrate this in HDNet, where they use normals to recover the intricate folds and wrinkles of clothing on human figures—details that are typically smoothed over or lost in standard depth maps.
Geometric Consistency as a Regularizer
Mathematically, surface normals are the normalized gradient of the depth map. If you know the depth of every pixel, you can derive the normals; conversely, if you know the normals, you can constrain the possible shapes the depth map can take.
When these two properties are estimated independently, they often contradict each other, leading to "piecewise flat" artifacts or noisy, physically impossible surfaces. Michael Strecke and Bastian Goldluecke show in Accurate Depth and Normal Maps from Occlusion-Aware Focal Stack Symmetry that by coupling these variables in a unified regularization framework, the system enforces physical consistency. The normals act as a "geometric regularizer," ensuring that the reconstructed surface follows learned local curvatures rather than just fitting noisy pixel matches.
Symbiotic Optimization
The most powerful aspect of this research direction is the "symbiotic relationship" created during optimization. In A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo, Yi Wei and his team describe an iterative process where depth and normals correct each other.
In this framework:
- Depth informs Normals: Initial depth estimates provide a coarse starting point for surface orientation.
- Normals refine Depth: The predicted normals are used to perform local plane fitting, which "hallucinates" correct depth in textureless regions where matching fails.
- Consistency Check: The system penalizes discrepancies between the depth gradient and the predicted normal vector.
This approach is particularly effective for challenging geometries. For instance, Minjun Kang and Jaesung Choe applied this joint learning to facial reconstruction using dual-pixel sensors. By forcing the model to predict both properties, they achieved smooth, realistic facial transitions even with the extremely narrow baselines found in smartphone cameras.
Beyond Euclidean Space
Recent work has also reimagined how these properties are represented to improve optimization stability. Jie Guo and Quewei Li suggest in Deep Surface Normal Estimation on the 2-Sphere that predicting normals as coordinates on a 2-sphere (angles) rather than 3D vectors allows for more precise "repair" of noisy depth data. Similarly, Ziyun Wang explores using Higher Order Function networks to predict the parameters of local tangent planes, effectively treating the surface as a continuous function rather than a discrete set of points.
By treating depth and normals as an inseparable pair, these researchers have moved 3D reconstruction away from simple distance-finding toward a more holistic understanding of scene geometry. The result is a generation of models that are not only more accurate but also more robust to the ambiguities of the real world.