When humans look at a two-dimensional image, we do not merely calculate distances based on stereoscopic disparities or local gradients. We understand what we are looking at. If we see a car, we know its approximate physical dimensions; if we see a shadow, we know it is an illumination change rather than a physical hole in the ground.
Traditional computer vision approaches often treat depth estimation as a purely low-level geometric or statistical problem. Whether through stereo matching or monocular regression, these systems attempt to map local pixel configurations directly to continuous depth values. However, this purely geometric formulation suffers from fundamental ambiguities. By integrating high-level semantic priors into low-level spatial reasoning, modern depth estimation frameworks resolve these contradictions, allowing models to infer scale, sharpen object boundaries, and fill in missing data in ill-posed regions.
Resolving Scale and Geometric Ambiguity
Monocular depth estimation inherently suffers from scale ambiguity: a small object close to the camera can project the exact same visual features as a large object far away. Purely spatial models struggle to differentiate between these scenarios without an understanding of scene context.
Purely Geometric View: [ Visual Feature ] ---> Ambiguous Scale (Close/Small vs. Far/Large)
Semantic-Guided View: [ Visual Feature + "Sedan" Category ] ---> Known Physical Size Prior ---> Absolute Scale
To overcome this, researchers have leveraged semantic categories as explicit size constraints. For instance, Lubor Ladicky et al. demonstrated in Pulling Things out of Perspective that performing semantic segmentation and depth estimation simultaneously allows a system to exploit the strict correlation between an object's semantic class and its expected physical dimensions. This insight has been scaled up by modern architectures; Ruijie Zhu et al. in ScaleDepth: Decomposing Metric Depth Estimation into Semantic-aware Scale Prediction and Adaptive Relative Depth Estimation utilized semantic embeddings from foundation models to predict a global scene-level scale factor, proving that common-sense knowledge about scene categories prevents physically implausible depth predictions.
Furthermore, semantic context helps models navigate deceptive visual environments. As shown by Lihe Yang et al. in Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data, aligning depth features with a high-level semantic encoder ensures that the model inherits an understanding of object categories. This prevents the network from misinterpreting surface textures, reflections, or shadows as physical changes in depth.
Boundary Refinement and Edge Preservation
A recurring failure mode in geometric depth estimation is the "bleeding" of depth values across object boundaries. When a sparse depth map (such as one from a LiDAR sensor) is upsampled, or when a stereo network computes matching costs, local color gradients are often used to find edges. However, color changes do not always correspond to depth discontinuities—a logo on a flat t-shirt or a change in pavement color can trick a model into inferring a non-existent physical cliff.
Color Gradient (Unreliable): [ Text / Logo on Wall ] ---> False Depth Discontinuity
Semantic Boundary (Reliable): [ Wall Category Boundary ] ---> Sharp, Accurate Depth Edge
High-level semantic masks provide a cleaner, category-specific prior for where depth should and should not change. Nick Schneider et al. proved in Semantically Guided Depth Upsampling that using semantic labels to guide depth propagation is significantly more reliable than relying on pixel intensities. It prevents depth values from bleeding across distinct objects because the network understands that pixels belonging to the same semantic instance should share a continuous geometric structure.
This interactive refinement is often implemented via joint multi-task learning. In Parallel Multi-Scale Semantic-Depth Interactive Fusion Network for Depth Estimation, Chenchen Fu et al. showed that allowing semantic features to interactively fuse with depth features across multiple scales directly improves edge definition around complex object boundaries. Similarly, Yukang Gan et al. demonstrated in Monocular Depth Estimation with Affinity, Vertical Pooling, and Label Enhancement that category-specific priors dictate how surface properties change, providing a structural rulebook for depth refinement networks.
Filling the Blanks in Ill-Posed Regions
In stereo matching and multi-view stereopsis, networks rely on finding corresponding pixels between different views. When a scene contains textureless regions (like a blank white wall), specular reflections (like a window), or repetitive patterns, local pixel matching fails entirely because the cost volume becomes highly ambiguous or multimodal.
Stereo Cost Volume (Textureless Wall): [ Multiple Weak Matches ] ---> Geometric Failure
Semantic Structural Prior: [ "Wall" = Vertical Plane ] ---> Inferred Smooth Depth
By introducing semantic context, the network can "fill in the blanks" where geometry fails. If a model recognizes a region as a "road" or a "building sky-line," it can apply learned structural priors about those categories to constrain the search space. Junming Zhang et al. demonstrated this synergy in DispSegNet: Leveraging Semantics for End-to-End Learning of Disparity Estimation from Stereo Imagery, showing that an unsupervised model can resolve depth in ill-posed, low-texture regions by tightly coupling disparity estimation with semantic segmentation.
On the multi-view side, the single-author work by Po-Han Huang, DeepMVS: Learning Multi-View Stereopsis, integrated high-level semantic features directly into the reconstruction pipeline, enabling the network to infer the geometry of specular and thin structures based on category context rather than raw pixel matching. This concept dates back to early joint estimation frameworks, such as the multi-modal auto-encoders proposed by Anthony R. Dick et al. in Multi-modal Auto-Encoders as Joint Estimators for Robotics Scene Understanding, which proved that coarse semantic labels provide a stronger constraint for spatial reconstruction than sparse or noisy geometric data alone.
Relational and Common-Sense Priors
Beyond simple category labels, the highest level of semantic integration involves understanding the logical relationships between objects in a scene. For example, if an object is classified as a "monitor" and another as a "desk," and the semantic relationship is "on top of," this relationship dictates strict spatial boundaries and depth constraints.
In Relationship Spatialization for Depth Estimation, Xiaoyu Xu formalized this by converting abstract scene graphs into spatially aligned visual features, demonstrating that even non-spatial semantic relationships provide critical depth priors that boost the performance of state-of-the-art models. By injecting these biological and relational cues—often via language embeddings and instance segmentation as explored by Dylan Auty et al. in Monocular Depth Estimation Using Cues Inspired by Biological Vision Systems—modern depth estimation systems move closer to human-like spatial reasoning, proving that knowing what we see is fundamental to understanding where it is.
Key Foundational Work
Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images (2013) — contributed 1 concept elements to this direction.
Recognition and Depth Perception of Objects in Real World Scenes (1977) — contributed 1 concept elements to this direction.
Priors for Stereo Vision under Adverse Weather Conditions (2013) — contributed 1 concept elements to this direction.