Resolving Monocular Scale Ambiguity with Stereo Training Constraints

15 concepts9 papers1 proposal

When humans look at a photograph, we easily infer the three-dimensional structure of the scene. We know that a car in the distance is large and far away, rather than a toy car positioned inches from our eyes. We achieve this by leveraging semantic priors—our lifelong knowledge of how big objects typically are.

For deep neural networks trained to estimate depth from a single image, however, this task is fundamentally ill-posed. The projection of a 3D scene onto a 2D image plane collapses depth up to an arbitrary scale factor. A network trained purely on monocular video sequences can learn to predict relative depth (which objects are closer than others), but it cannot inherently know whether a distance represents 5 meters or 50 meters.

To overcome this absolute scale ambiguity without relying on expensive ground-truth sensors like LiDAR, a powerful paradigm has emerged: using stereo camera geometry during training to anchor absolute physical scale, while maintaining a strictly monocular profile during inference.

The Core Mechanism: Stereo as a Scale Anchor

To understand why this approach works, we must look at the geometric constraints of a stereo camera pair. A stereo rig consists of two cameras separated by a fixed, known horizontal distance, called the baseline ($b$).

When a network is trained to predict depth from the left image ($I_L$), we can use its predicted depth map ($\hat{Z}$) along with the known baseline $b$ and camera focal length ($f$) to calculate the horizontal disparity ($d = bf / \hat{Z}$) for every pixel. This disparity allows us to mathematically warp the right image ($I_R$) to reconstruct the left image.

$$\hat{I}_L(x, y) = I_R(x - d, y)$$

If the network's depth prediction $\hat{Z}$ is incorrect by some scale factor $\alpha$, the calculated disparity will be scaled incorrectly, causing the warped image $\hat{I}_L$ to mismatch the actual image $I_L$. By minimizing the photometric reconstruction error between $\hat{I}_L$ and $I_L$, the network is mathematically forced to predict depth in absolute physical units that match the baseline $b$.

As Zhan et al. (2019) demonstrate, this spatial consistency constraint allows self-supervised models to learn real-world units without requiring any human-labeled ground truth. Crucially, because the stereo pair is only used to compute the loss during training, the second camera can be discarded entirely during deployment.

Training Phase (Stereo Rig):
[Left Image]  --> [ Monocular Depth Network ] --> [Predicted Metric Depth] 
                                                          |
                                               (Warp using known baseline 'b')
                                                          |
[Right Image] ----------------------------------> [Photometric Loss]

Inference Phase (Single Camera):
[Single Image] --> [ Monocular Depth Network ] --> [Absolute Metric Depth]

Why It Generalizes to Monocular Inference

A common point of confusion is why a network trained this way can still output absolute scale when the right camera is removed during inference. If the geometric baseline $b$ is missing at runtime, how does the model know the scale?

The answer lies in the network's ability to bind semantic features to geometric realities. During training, the photometric loss forces the network to output metric-accurate depth. To minimize this loss across thousands of training images, the network learns to associate the visual appearance and scale of known objects—such as the width of a lane marking, the height of a sedan, or the texture gradient of the road—with their corresponding physical distances.

As explored by Vankadari et al. (2019), incorporating stereo image pairs during training allows the model to internalize this absolute scale, effectively overcoming the inherent scale ambiguity of single-camera frameworks. At inference time, the network acts as a sophisticated estimator that maps these learned semantic and structural cues directly to metric units.

Scaling to Motion: Odometry and Scene Flow

The benefits of stereo-supervised scale recovery extend far beyond static depth maps; they fundamentally stabilize the estimation of camera ego-motion and scene dynamics over time.

In purely monocular visual odometry (VO), scale drift is a catastrophic problem. Because the scale factor can change arbitrarily from frame to frame, the estimated trajectory of a vehicle quickly warps and accumulates massive errors. By introducing a fixed spatial reference during training, the network learns to output velocities and translations in consistent physical dimensions.

Several distinct frameworks leverage this property:

  • Visual-Inertial Fusion: Han et al. (2019) combine monocular video with Inertial Measurement Unit (IMU) data. By using stereo-derived 3D geometric constraints during self-supervised training, their framework, DeepVIO, enables a single-camera system to recover absolute physical scale and maintain highly robust trajectory estimates in messy real-world conditions.
  • Online Trajectory Correction: Zhang et al. (2021) utilize stereo imagery during training to ensure that monocular visual odometry models maintain consistent physical dimensions during real-world deployment, preventing the scale drift that plagues traditional monocular pipelines.
  • Holistic 3D Motion Parsing: Rather than treating depth and motion as isolated problems, Yang et al. (2018) integrate stereo constraints into a holistic framework that models depth, optical flow, and camera pose simultaneously. The stereo baseline serves as a sufficient geometric anchor to ensure that both distances and velocities are resolved without scale ambiguity.
  • Dynamic Scene Flow: Hur et al. (2020) extend this concept to predict the full 3D motion of every point in a scene (scene flow) from consecutive monocular frames, relying on stereo pairs exclusively during training to anchor the 3D coordinate space.

Temporal and Spatial Cross-Supervision

To maximize the accuracy of these models, researchers often combine spatial consistency (left-to-right stereo matching) with temporal consistency (frame-to-frame video matching).

As shown by Zhan et al. (2018), training on stereo video sequences provides a dual-constraint optimization problem. The temporal frames teach the network about motion and continuity, while the stereo baseline locks the absolute scale. This dual-supervision prevents the network from taking shortcuts, resulting in highly accurate metric depth and visual odometry.

Similarly, Andraghetti et al. (2019) demonstrate that traditional, sparse visual odometry can be integrated alongside stereo spatial consistency constraints. The sparse geometric points act as an additional anchor, allowing even lightweight neural architectures to achieve high-fidelity metric depth estimation.

Alternative Anchors: Sparse Point Distillation

While using a full stereo image pair during training is the most common approach, the underlying principle—anchoring relative semantic predictions to absolute geometric constraints—can be inverted or sparsified.

For instance, Tosi et al. (2020) explore a variation where a monocular network is provided with a sparse set of high-reliability geometric depth points as an additional input channel. Instead of relying on a dense second image, the network learns to align its dense semantic understanding of the scene structure with these sparse absolute physical dimensions. This demonstrates that the core mechanism of scale recovery does not strictly require dense stereo pairs, but rather any reliable geometric constraint that can act as a metric anchor during the learning process.

Conclusion

The power of stereo-to-monocular scale transfer lies in its asymmetric design. It acknowledges that while stereo cameras or geometric sensors are cumbersome and expensive to deploy at scale in production, they are readily available in laboratory or data-collection environments. By using geometric constraints to ground a network's semantic understanding during training, models can perform inference with the simplicity of a single camera and the metric precision of a multi-sensor rig.

Key Foundational Work

Unsupervised Learning of Monocular Depth and Ego-Motion using Conditional PatchGANs (2019) — contributed 2 concept elements to this direction.
Unsupervised Learning of Monocular Depth Estimation and Visual Odometry With Deep Feature Reconstruction (2018) — contributed 2 concept elements to this direction.
Every Pixel Counts: Unsupervised Geometry Learning with Holistic 3D Motion Understanding (2018) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stereo_to_monocular_scale_transfer_470d7176,
  title = {stereo-to-monocular scale transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stereo-to-monocular-scale-transfer/rdr_470d71765d8a4bbfac85e58ef909eead},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_stereo_to_monocular_scale_transfer_470d7176_v1,
  title = {stereo-to-monocular scale transfer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stereo-to-monocular-scale-transfer/rdr_470d71765d8a4bbfac85e58ef909eead/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake