When we map the 3D world onto a 2D image plane, the physics of perspective projection introduces a fundamental non-linearity: the perceived size and motion of an object are inversely proportional to its distance from the camera. Despite this, early computer vision models often treated depth uniformly, allocating equal representation capacity or sampling steps to an object at 1 meter as one at 100 meters.
This uniform approach introduces severe inefficiencies. Near-field regions become undersampled, causing geometric blur or matching failures because small physical movements translate to massive pixel shifts. Meanwhile, far-field regions waste computational budget on sub-pixel details that cannot be resolved by the sensor. To bridge this gap, modern 3D vision relies on perspective-aware depth sampling and representation strategies that align computational capacity with the physics of imaging.
The Geometry of Inverse Depth and Disparity
To align the computational search space with camera optics, multi-view stereo (MVS) and neural rendering frameworks frequently shift from linear depth ($z$) to inverse depth ($1/z$), which is linearly proportional to pixel disparity.
As shown by Xu et al. in Learning Inverse Depth Regression for Multi-View Stereo with Correlation Cost Volume, sampling depth hypotheses uniformly in inverse depth space ensures that the projected points are distributed evenly along epipolar lines in neighboring views. If a system samples uniformly in linear depth, the search points become overly crowded near the camera and too sparse far away, degrading matching robustness at the extremes of the depth range. Uniform inverse-depth sampling ensures sub-pixel matching accuracy for both near and far objects.
Linear Depth Sampling (Wastes far-field capacity):
|----|----|----|----|----|----|----|----|----|----|
Near Far
Inverse Depth Sampling (Matches pixel parallax):
|-------|-----|----|---|---|--|--|-|-|
Near Far
This geometric insight is equally vital for implicit neural representations. In Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields, Barron et al. demonstrated that distributing ray samples linearly in inverse-distance space prevents the background collapse typical of unbounded scene reconstructions. Because the perceived size of an object decreases with distance, sampling proportionally to inverse distance ensures that background regions receive a sampling density proportional to their visual footprint in the final rendered image.
Logarithmic Discretization and Scale Invariance
While inverse depth perfectly models the geometry of multi-view triangulation, representing depth in logarithmic space ($\log z$) provides an alternative that guarantees scale invariance and a constant relative error.
The foundational mathematical insight that perspective image irradiance equations are inherently tied to the derivatives of the natural logarithm of depth was explored early on by Tankus et al. in A New Perspective [on] Shape-from-Shading. Similarly, Huang et al. demonstrated in Statistics of Range Images that a log-range transformation keeps the statistical properties of 3D shapes invariant to their distance from the sensor.
In modern monocular depth estimation, treating depth as a continuous linear regression target often causes networks to overfit to large outdoor distances, compressing and ignoring fine-grained foreground structures. To combat this, Cao et al. in Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks and Fu et al. in Deep Ordinal Regression Network for Monocular Depth Estimation reframed depth prediction as a classification or ordinal regression task over non-uniform, logarithmically spaced bins.
This "spacing-increasing discretization" strategy prioritizes spatial resolution in the near-field—where precision is critical for robotics, navigation, and interaction—while naturally accommodating the increasing measurement uncertainty at greater distances.
This log-space parameterization also aids generative models. Saxena et al. utilized a log-scale depth representation in Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model to allow a single diffusion model to allocate sufficient capacity to both tight indoor spaces and expansive outdoor environments. As noted by the authors of From Editor to Dense Geometry Estimator, logarithmic quantization ensures that relative error remains nearly constant across varying distances, bypassing the precision limitations of standard floating-point formats where distant objects suffer from disproportionately large geometric errors.
Jacobian-Based and Adaptive Space Warping
For complex camera trajectories or non-standard sensor geometries, simple inverse or log transformations may not fully capture the scene's perspective structure. In these scenarios, the optimal sampling density can be mathematically derived from the camera's perspective mapping using coordinate warping functions.
In F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories, Wang et al. introduced a generalized perspective warping technique where the ray sampling strategy is explicitly derived from the Jacobian of a spatial warping function. By aligning the 3D ray sampling density with the 2D image resolution through this Jacobian, the model ensures that the most visually significant parts of an arbitrary camera trajectory receive the highest sampling priority, drastically accelerating training convergence.
[3D Space Coordinates] ---> (Warping Function) ---> [Perspective Space]
|
(Take Jacobian Matrix)
|
v
[Optimal Ray Sampling Density]
When dealing with non-pinhole optics, such as wide-field-of-view or fisheye lenses, standard linear or inverse-depth sampling fails because pixels are distributed non-linearly across a spherical geometry. To address this, the authors of FoVA-Depth: Field-of-View Agnostic Depth Estimation for Cross-Dataset Generalization sampled depth hypotheses based on the reciprocal tangent of the viewing angle. This strategy aligns the search space directly with the non-linear pixel distribution of spherical geometries, demonstrating that perspective-aware sampling is essential for cross-camera generalization.
Why It Works: A Unified Intuition
Whether through inverse depth, logarithmic binning, or Jacobian-based space warping, the underlying principle remains identical: the representation space must mirror the measurement physics.
Linear depth representations treat space from the perspective of the world, whereas cameras perceive space from a single focal point. By compressing the representation of the distant background and expanding the resolution of the foreground, perspective-aware sampling ensures that every sample, cost-volume bin, or network parameter contributes optimally to the final 2D projection. This mathematical alignment resolves the tension between infinite real-world scales and finite computational budgets, making modern 3D vision systems both highly accurate and computationally viable.
Key Foundational Work
A New Perspective [on] Shape-from-Shading (2003) — contributed 3 concept elements to this direction.
Statistics of Range Images (2000) — contributed 1 concept elements to this direction.