Learning 3D Geometry through Relative Depth Ranking

178 concepts85 papers1 proposal

When computer vision models attempt to reconstruct the 3D world from 2D images, they face a fundamental mathematical hurdle: monocular depth estimation is inherently ill-posed. A small object close to the lens projects the exact same image as a massive object far away.

Traditional deep learning approaches try to solve this by treating depth estimation as a standard regression problem, forcing networks to predict absolute metric distances (e.g., determining that a specific pixel is exactly 3.4 meters away). However, absolute regression is highly sensitive to camera intrinsics, focal lengths, and global scale factors. A model trained on one camera setup frequently fails on another due to scale and shift inconsistencies. Furthermore, collecting dense, absolute ground-truth depth requires expensive hardware like LiDAR or synthetic environments, which introduces a massive domain gap, as noted in Play and Learn: Using Video Games to Train Computer Vision Models.

In contrast, human vision rarely calculates exact metric distances. Instead, our visual system relies on relative depth relationships—we instantly perceive that object A is in front of object B, or that a person's hand is closer than their shoulder. Shifting the learning objective from absolute regression to relative depth ranking (or ordinal depth supervision) allows models to bypass scale ambiguities entirely and capture the invariant geometric structure of a scene.

From Metrics to Orderings: How Ranking Losses Work

To enforce relative depth mathematically, models are trained using ranking losses rather than mean squared error (MSE). Instead of penalizing a network for being off by a few centimeters, a ranking loss penalizes the network if its predicted depth order contradicts the true front-to-back relationships in the scene.

As pioneered by Weifeng Chen in Single-Image Depth Perception in the Wild, a model can learn continuous, high-quality depth maps from unconstrained internet imagery without any metric ground truth. By sampling pairs of points from an image and optimizing a pairwise ranking loss based on qualitative human judgments (i.e., "point A is closer than point B"), the network learns to map visual features directly to relative spatial arrangements.

Pairwise Sampling:
  Point A (Foreground) vs Point B (Background)
  Loss penalizes if Predicted_Depth(A) >= Predicted_Depth(B)

While pairwise comparisons are highly effective, evaluating every possible pair of pixels introduces quadratic computational complexity and can miss broader scene context. To capture global transitivity and accelerate training, Eyke Hüllermeier et al. introduced a listwise ranking approach in Monocular Depth Estimation via Listwise Ranking Using the Plackett-Luce Model. Instead of treating depth as a series of isolated pairs, the authors frame depth estimation as a permutation problem across entire sets of points.

By optimizing for the correct sequence of multiple points simultaneously, listwise ranking prevents local depth inversions and ensures global structural consistency. Because these ranking objectives focus entirely on the order of pixels rather than their absolute values, they avoid the smoothing artifacts typical of MSE regression, preserving sharp object boundaries and fine-grained structural details.

Resolving Geometric Ambiguities in 2D-to-3D Lifting

The power of relative depth ranking becomes especially apparent when "lifting" 2D projections into 3D space, such as in human pose estimation or multi-object tracking. Lifting a 2D skeleton to 3D suffers from a severe "mirroring ambiguity"—a limb can bend forward or backward while projecting to the exact same 2D coordinates on the image plane.

The authors of DRPose3D: Depth Ranking in 3D Human Pose Estimation demonstrated that transforming absolute joint depth regression into a series of pairwise classification tasks provides a much more stable learning objective. When a model is forced to predict the relative z-order of joints (e.g., whether the left wrist is closer to the camera than the right shoulder), the geometric ambiguity collapses. Once the relative ordering along the camera axis is established, the underlying 3D skeletal structure becomes mathematically well-defined.

This qualitative constraint makes weak supervision incredibly cost-effective. Pietro Perona showed in It's all Relative: Monocular 3D Human Pose Estimation from Weakly Supervised Data that sparse, crowd-sourced relative annotations, when combined with basic geometric constraints like constant limb lengths, allow models to achieve accuracy levels nearing those of fully supervised systems. Similarly, Saurabh Sharma utilized ordinal joint relations as a robust selection criterion in Monocular 3D Human Pose Estimation by Generation and Ordinal Ranking, scoring and filtering multiple physically plausible 3D pose hypotheses generated by a variational autoencoder.

2D Keypoints ---> CVAE Generates Multiple 3D Hypotheses ---> Ordinal Rank Filter ---> Coherent 3D Pose

In multi-person scenes, relative depth constraints prevent physical impossibilities, such as depth-wise collisions or subjects "clipping" through one another. Nikos Kolotouros et al. leveraged 2D instance segmentation masks as a proxy for 3D depth supervision in Coherent Reconstruction of Multiple Humans From a Single Image. By using differentiable rendering to project 3D human meshes back onto the image plane, the authors calculated an ordinal loss that compared the rendered depth order against 2D occlusion boundaries. This constraint forces the model to correctly position subjects in 3D space based entirely on who is blocking the view of whom.

Anchoring Sparse-View and Generative 3D Reconstructions

Beyond single-image estimation, relative depth ranking has become a vital tool for regularizing novel view synthesis methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). When optimizing these models from sparse viewpoints (e.g., only three or four images), the optimization problem is highly underconstrained. Generative models and neural renderers often collapse into degenerate "billboard" solutions—where objects appear as flat 2D planes—or generate floating artifacts to satisfy the sparse views.

To anchor the geometry, researchers frequently turn to large-scale monocular depth foundation models to provide geometric priors. However, because monocular models suffer from severe scale drift and temporal inconsistencies across different camera angles, forcing a 3D scene to match absolute monocular depth values causes the optimization to fail due to conflicting multi-view scales.

The solution is to supervise the 3D model using intra-view relative depth ranking rather than absolute values. Dejia Xu et al. implemented this strategy in NeuralLift-360: Lifting an in-the-Wild 2D Photo to A 3D Object with 360° Views. By applying a ranking loss, they ensured that the relative ordering of pixels rendered from a specific viewpoint matched the monocular prior, without locking the underlying 3D volume into an incorrect absolute scale.

Monocular Prior (Scale Inconsistent) ---> Extract Relative Ranks ---> Supervise NeRF/3DGS via Ranking Loss (Scale Invariant)

This local patch-based optimization is highly robust to noise. In RDG-GS: Relative Depth Guidance with Gaussian Splatting for Real-time Sparse-View 3D Rendering, Chenlu Zhan et al. optimized for relative depth relationships within local image patches. This approach ensures that neighboring regions maintain consistent spatial structures across viewpoints, preventing geometric conflicts between different camera perspectives.

As Han Huang demonstrated in FatesGS: Fast and Accurate Sparse-View Surface Reconstruction Using Gaussian Splatting with Depth-Feature Consistency, integrating monocular depth priors through an intra-view ranking loss allows the 3D reconstruction to adapt dynamically to the true scene geometry while fully exploiting the strong ordinal accuracy of foundation models. By focusing on the invariant "front-to-back" relationships, ordinal depth supervision provides a robust geometric scaffold across a wide array of computer vision applications.

Key Foundational Work

Shapecollage: Occlusion-Aware, Example-Based Shape Interpretation (2012) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 94 papers
  2. Version 1 · 2026-10-04 · 129 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_ordinal_depth_supervision_7fb78f73,
  title = {ordinal depth supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ordinal-depth-supervision/rdr_7fb78f7335634444afabff331b7454f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_ordinal_depth_supervision_7fb78f73_v2,
  title = {ordinal depth supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ordinal-depth-supervision/rdr_7fb78f7335634444afabff331b7454f0/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake