Monocular depth estimation models have achieved remarkable success by leveraging deep neural networks to infer 3D structure from a single 2D image. However, these models suffer from a fundamental mathematical limitation: they are inherently affine-invariant. Because a single lens cannot determine absolute physical size without external context, monocular networks predict depth up to an arbitrary scale factor ($s$) and a global translation or shift ($t$).
When applied frame-by-frame to a video sequence, this ambiguity introduces severe challenges. A model might predict an entirely different scale for consecutive frames, causing "scale drift" and temporal flickering. Furthermore, when integrating these predictions with physical sensors (such as LiDAR or radar) or geometric algorithms (such as Structure-from-Motion or stereo matching), the lack of a stable metric scale makes direct fusion impossible.
To resolve this, researchers have developed scale alignment techniques. The core intuition is to treat the raw monocular depth map as a structurally accurate but uncalibrated geometric prior, and then compute the optimal transformation parameters to map it into a globally coherent or physically accurate metric coordinate system.
Temporal Alignment: Enforcing Consistency Across Time
To prevent a model from changing its internal "ruler" from frame to frame in a video sequence, self-supervised learning frameworks leverage geometric constraints across adjacent views. The underlying mechanism relies on differentiable warping. If the camera's ego-motion between two frames is known or estimated, the depth map of Frame A can be projected into 3D space, transformed by the camera's trajectory, and re-projected into the viewpoint of Frame B. If the model's scale is consistent over time, this warped depth map should match the direct depth prediction of Frame B.
Jia-Wang Bian et al. formalized this intuition in Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video by introducing a geometry consistency loss. This loss explicitly penalizes pixel-wise scale discrepancies between the warped and predicted depth maps, forcing the network to maintain a uniform scale across the entire video sequence. In subsequent work, Jia-Wang Bian demonstrated in Unsupervised Scale-Consistent Depth Learning from Video that enforcing this scale uniformity allows a standard monocular camera to function similarly to an RGB-D sensor, providing the stable initialization required for robust Visual SLAM systems.
To make this temporal alignment resilient to lighting changes and dynamic objects, various reference anchors are utilized:
- Mean Depth Anchoring: Tianwei Shen et al. enforced scale alignment by using the mean depth of adjacent frames as a stable reference during training, preventing geometric drift even when standard photometric consistency assumptions fail.
- Decoupled Geometric Loss: Lijun Wang et al. introduced a Scale-Aware Geometric (SAG) loss that decouples object appearance from spatial scale, ensuring that ego-motion trajectories remain stable and match real-world movement.
- Bidirectional Regularization: Heming Du et al. utilized bidirectional geometry consistency regularization to stabilize single-lens depth predictions across frames, providing a reliable spatial representation for robotic navigation.
- Persistent Anchors: To eliminate cumulative scale drift over long trajectories, the authors of Flow4R proposed generating local 3D maps where the initial frame serves as a persistent scale anchor for all subsequent frames in the sequence.
Cross-Modal Alignment: Anchoring to Metric Reality
While temporal alignment ensures that depth maps are consistent relative to each other, it does not provide absolute metric accuracy. To ground qualitative depth maps in real-world units (such as meters), the scale-agnostic predictions must be aligned with sparse but highly accurate geometric data from physical sensors or classical multi-view geometry.
Classical algorithms like Structure-from-Motion (SfM) and stereo matching provide exact metric points based on rigid triangulation, but they often fail in textureless regions or under occlusions. Conversely, monocular foundation models provide dense, continuous depth maps but lack scale. Cross-modal alignment bridges this gap by fitting a mathematical transformation—typically via least-squares regression or Random Sample Consensus (RANSAC)—to align the dense relative depth with the sparse metric anchors.
[Dense, Scale-Agnostic Monocular Depth] ---\
---> [Least-Squares / RANSAC] ---> [Dense, Scale-Accurate Metric Depth]
[Sparse, Metric Anchors (SfM / LiDAR)] ----/
For instance, Haoyu Guo et al. developed a pipeline that uses a RANSAC-based linear regressor to align diffusion-based monocular depth predictions with sparse SfM point clouds, ensuring that the dense outputs are globally consistent across multiple views. Similarly, Jaeyoung Chung et al. resolved scale ambiguity by calculating scale and offset parameters from SfM points, transforming relative depth maps into a dense geometric guide for few-shot 3D Gaussian Splatting. For general out-of-distribution depth completion, Shenglun Chen et al. demonstrated that precise metric calibration via least-squares alignment provides significantly greater stability than heuristic scaling methods.
This optimization strategy applies directly to hardware sensor fusion:
- Radar: Han Li et al. utilized global scale alignment to synchronize relative monocular depth maps with sparse metric data from radar, providing a physically grounded foundation for pixel-wise depth refinement in adverse weather.
- LiDAR: Bharat Singh used optimization techniques to align relative depth maps with background LiDAR points, converting qualitative predictions into precise metric measurements for 3D sensor simulation.
- Tactile Sensors: Aiden Swann et al. introduced a hierarchical approach that corrects monocular scale ambiguity by anchoring the visual depth map to high-precision physical contact measurements.
Learnable Alignment in Neural Radiance Fields
In modern 3D reconstruction frameworks like Neural Radiance Fields (NeRF) and Signed Distance Functions (SDF), monocular depth maps are frequently used as geometric priors to guide optimization, particularly in featureless or unposed environments. However, forcing a neural network to strictly match an uncalibrated, frame-to-frame inconsistent depth prior introduces geometric distortions.
Instead of treating the monocular depth as an absolute target, modern architectures incorporate the scale ($s$) and shift ($t$) parameters directly into the optimization loop as learnable variables. As the network optimizes for color and geometry, it simultaneously learns the optimal affine transformation for each frame's depth map to align it with the evolving global 3D coordinate system.
$$\text{Depth}_{\text{aligned}} = s_i \cdot \text{Depth}_{\text{monocular}} + t_i$$
Wenjing Bian et al. implemented this in NoPe-NeRF, assigning learnable transformation parameters to each frame's depth map. This formulation allows the model to dynamically adjust the depth prior during joint pose and scene optimization, distilling inconsistent frame-to-frame predictions into a coherent global geometry.
Similarly, Zehao Yu et al. incorporated learnable per-batch parameters in MonoSDF to align relative monocular depth with rendered absolute depth, ensuring that the geometric prior guides the spatial scale without introducing incorrect positional biases. This affine-invariant supervision strategy is also critical in specialized domains; Romain Hardy et al. applied affine-invariant depth supervision to align rendered depth with monocular predictions, enabling physically consistent 3D colonoscopic reconstructions from casual handheld videos.
Resolving Multi-Body and Stereo Ambiguities
Scale alignment also provides solutions for complex multi-body dynamics and zero-shot stereo matching. When a scene contains multiple independently moving objects, classical multi-view geometry struggles because each moving object appears to have its own arbitrary scale factor.
Andrea Porfiri Dal Cin et al. resolved this relative scale ambiguity by using monocular depth priors as a global scale reference. By comparing the triangulated geometry of moving objects against the learned depth priors, they estimated a consistent scale factor to unify independent rigid bodies into a single coordinate system. For dynamic view synthesis, Jae Shin Yoon et al. used the static background of a scene as a scale anchor, normalizing the depth values of dynamic foreground objects into a globally coherent 3D coordinate system.
In the context of stereo matching, scale alignment helps integrate monocular foundation models to resolve ambiguities in occluded or textureless zones. Fabio Tosi et al. introduced a differentiable module that uses geometric stereo features to estimate the optimal scale and shift parameters for monocular contextual priors, ensuring they map correctly to the metric geometry of the stereo scene.
To handle substantial initialization errors, Hualie Jiang et al. utilized global search operations on a correlation volume to identify the optimal scale factor. This global approach allows the model to recover from significant initial scale discrepancies that local search windows would fail to resolve, combining the contextual strength of monocular models with the rigid precision of stereo geometry.
Key Foundational Work
Self-Supervised Learning of Depth and Motion Under Photometric Inconsistency (2019) — contributed 3 concept elements to this direction.
Self-supervised Monocular Depth and Visual Odometry Learning with Scale-consistent Geometric Constraints (2020) — contributed 2 concept elements to this direction.
Consistent video depth estimation (2020) — contributed 1 concept elements to this direction.