In traditional computer vision, researchers often divide the problem of scene understanding into two distinct silos: "what" is in the image (semantics) and "where" it is in 3D space (geometry). This decoupling treats semantic segmentation and depth estimation as independent regression tasks. However, the physical world does not operate in silos. A "road" is almost always a horizontal plane; a "building" is a vertical one. By treating these tasks as unrelated, models often produce physically impossible predictions, such as vertical roads or depth maps that ignore the boundaries of objects.
The emerging research direction across these 91 papers focuses on jointly modeling geometry and semantics to enforce physical consistency. By mathematically linking these tasks, models can use semantic priors to regularize geometry and use geometric constraints to refine semantic labels.
The Geometry of Motion and Depth
The most fundamental link in this field is the relationship between depth, optical flow, and camera pose. As Yuliang Zou and colleagues argue in DF-Net, these three elements are inextricably linked by the laws of projective geometry. If you know how the camera moved (pose) and how far away a pixel is (depth), you can mathematically determine where that pixel should appear in the next frame (flow).
GeoNet exploits this by using an unsupervised consistency loss. Instead of learning three separate functions, the model learns to predict them such that they satisfy the rigid-body transformation equations. This "mutual supervision" allows the model to resolve ambiguities—for instance, using the camera's motion to better estimate the depth of low-texture regions where standard stereo matching might fail.
Semantics as a Geometric Prior
While geometry provides hard mathematical constraints, semantics provide powerful statistical priors. Stephen Gould, Richard Fulton, and Daphne Koller pioneered this by decomposing scenes into semantically consistent regions that share geometric orientations.
The intuition is simple: if a network identifies a region as a "sky," it should not assign it a depth of five meters. Peng Wang and Brian L. Price expanded on this in Towards unified depth and semantic prediction, showing that semantic labels can act as a "global plan" for the depth map, ensuring that surfaces belonging to the same object maintain structural continuity. This is particularly effective at object boundaries, where Eunhyeok Park and Sungjoo Yoo use fine-grained semantics to sharpen depth transitions that would otherwise be blurred by standard convolutional filters.
Resolving Scale Ambiguity
One of the "holy grails" of monocular vision is recovering absolute metric scale—knowing exactly how many meters away an object is from a single image. Mathematically, this is impossible without a reference.
Recent work solves this by using known semantic entities as "rulers." Lea Müller and Jitendra Malik demonstrate this in HSfM by leveraging the known statistical height of the human body to recover the absolute scale of an entire 3D scene. Similarly, Haimei Zhao and Dacheng Tao introduced JPerceiver, which uses the known width of road lanes in driving scenes to anchor monocular depth estimates to a real-world metric scale.
Feature-Level Symbiosis
The most recent shift in this research direction is moving beyond shared loss functions toward internal feature sharing. Instead of having two separate "heads" on a shared backbone, researchers are building architectures where tasks "talk" to each other during the inference process.
- Distillation: Dan Xu and colleagues developed PAD-Net, which predicts auxiliary tasks like surface normals and then "distills" that geometric information into the semantic segmentation head.
- Attention: Nitin Bansal and Junsong Yuan proposed a symbiotic approach using Cross-Channel Attention Modules (CCAM). This allows the network to dynamically weigh which semantic features are most relevant for depth (e.g., edges) and which geometric features are most relevant for semantics (e.g., planarity).
By treating the scene as a single, unified entity where "what" and "where" are two sides of the same coin, these models achieve a level of structural consistency that independent models cannot match. The result is not just better numbers on a benchmark, but a 3D reconstruction that finally obeys the laws of the physical world.