A single photograph cannot tell a dollhouse from a house. If every distance in a scene is multiplied by the same factor, the image does not change. Monocular depth networks therefore learn relative shape reliably but have to guess absolute scale in meters. This proposal takes that fact literally. The image is unchanged under a group of uniform rescalings of the scene, and the metric depth map is not. Producing metric depth is therefore an instance of symmetry breaking: the output has less symmetry than the input. A result from equivariant learning (Equivariant Symmetry Breaking Sets) says that a network respecting a symmetry cannot break it on its own. It needs an explicit external "breaking object," and when the choice of that object is ambiguous, it should select from a small set of candidates rather than average over them.
We propose a monocular depth model built on this idea. It has two parts. The first is a shape network that is exactly blind to scale. The second is a scale module that is exactly equivariant to its physical anchors: if every anchor distance is multiplied by α, the predicted depth map is multiplied by exactly α. The anchors can be sparse LiDAR returns or visual-inertial points. When no anchors are available, the scale module instead produces a small set of image-derived scale hypotheses and is trained with a best-of-set loss. The claim to test is that this construction extrapolates to depth ranges never seen in training and degrades gracefully as anchors become fewer, noisier, or absent. Learned sparse-to-dense fusion networks and scale-jitter augmentation should do neither.
Core Claim
Write $D(p)$ for metric depth at pixel $p$ and work in log-depth, where uniform rescaling of the scene becomes adding a constant. The model predicts
$$\log D(p) = g(I)(p) + s(I, A),$$
where $g(I)$ is a per-pixel log-shape map from image $I$, normalized to zero median so it carries no scale, and $s$ is a single log-scale computed from image $I$ and the set of metric anchors $A$. The construction enforces two properties exactly. First, $g$ depends on the image only. Second, $s(I, \alpha A) = s(I, A) + \log \alpha$ for every $\alpha > 0$, which we get by canonicalizing: subtract a robust log-statistic of the anchors, process in normalized units, then add it back. This is canonicalization rather than an equivariant layer stack. That choice matters, because heavy equivariant architectures have lost to augmentation in a closely related monocular setting.
When $A$ is empty, the scale module outputs $K$ hypotheses $s_1, \dots, s_K$ with confidences. It is trained with
$$\mathcal{L} = \min_{k} \, \ell\big(g(I) + s_k,\ \log D^{*}\big) + \lambda \, \mathrm{CE}(\pi, k^{*}),$$
where $D^{*}$ is ground truth, $\ell$ is a per-pixel log-depth error, $\pi$ is the confidence distribution, and $k^{*}$ is the minimizing hypothesis. This is the "min-over-set" loss of the symmetry-breaking-set work, carried from crystal symmetries to camera scale. As soon as even one anchor is present, choosing among hypotheses becomes nearly trivial.
What is new is not "decompose depth into scale and shape," which ScaleDepth already does. It is three things together: exact equivariance of scale to physical anchors with a nonlinear learned correction; explicit multi-hypothesis scale when anchors are missing; and a test showing that these properties, rather than capacity, explain any out-of-range generalization.
Relation to Prior Work
Our closest relatives decompose scale from structure. ScaleDepth predicts metric depth as a global scale times a normalized relative map, regressing the scale from frozen CLIP text embeddings. Its authors note that this fails when scenes fall outside the prompt vocabulary. We keep the factorization but differ in two ways. Our scale is tied to physical anchors by an exact group relation rather than regressed. And without anchors we output a set of hypotheses rather than one value, so an ambiguous scene does not collapse to a blurred average scale.
Monocular Visual-Inertial Depth Estimation fits affine-invariant depth to sparse visual-inertial points by global least squares, then refines with a learned per-pixel scale map (ScaleMapLearner). The global fit is already scale-equivariant. The learned refinement, as described, is not constrained to be. Our model makes the whole nonlinear correction equivariant. It also drops the depth-shift parameter, because a single anchor then suffices.
LiDARTouch feeds a 4-beam LiDAR scan directly into a dense network. This removes the infinite-depth failure but leaves the network free to learn depth-range-specific behavior. Together, these two papers motivate the work, since they show that a few anchors suffice. They also define the competitor we must beat.
Metric3D v2 and ZeroDepth resolve the focal-length part of the ambiguity, Metric3D v2 by resampling to a canonical camera and ZeroDepth by embedding camera rays. We treat their contribution as orthogonal and reuse canonical-camera resampling inside $g$. A scale-equivariant architecture cannot recover focal length from pixels alone, so we do not pretend it can. The scene-scale ambiguity that remains after intrinsics are known is what our model addresses.
Two papers constrain the design. On the Role of Rotation Equivariance in Monocular 3D Human Pose Estimation finds that an unconstrained ResNet with rotation augmentation beats equivariant-by-design models: 64.3 mm versus 76.4 mm on Rotated Human3.6M, with roughly 30× faster training. SE(3) Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation shows that exact SE(3) equivariance helps in multi-view depth, but at about 2× training cost from high-order spherical harmonics. These results are why we enforce scale symmetry by cheap canonicalization on a one-dimensional group and leave the image backbone unconstrained. The equivariance-versus-augmentation tension is real, and we test it directly rather than assume it away.
Research Program
We build three components on a shared vision-transformer backbone with Metric3D-style canonical-camera resampling.
The first component is the shape head $g$. It is trained with a scale-invariant log-depth loss on mixed indoor and outdoor data.
The second is the anchored scale module. It encodes each anchor by its pixel location, its canonicalized log-depth, and the shape prediction at that pixel. It pools these with a set transformer and outputs a log-scale correction plus a low-resolution per-pixel residual, which is also computed in canonical units, so equivariance holds by construction. We will choose between median and trimmed mean for the canonicalizing statistic by robustness to outlier anchors.
The third is the hypothesis head. It produces $K$ log-scales from image features, and we will sweep $K$ from 1 to 8.
The analysis program asks three questions. Does exact equivariance buy range extrapolation that augmentation cannot? Does the hypothesis set capture genuinely multimodal scale, measured as a gap between best-of-K and single-prediction error that concentrates on scenes humans would also find ambiguous? And how does accuracy fall as the number of anchors goes from roughly 150, the visual-inertial regime, to 4 beams, to 1 point, to 0?
Baselines
The naive reference is a frozen relative-depth network aligned to anchors by global median or least-squares scaling, exactly the first stage of Monocular Visual-Inertial Depth Estimation. It is exactly equivariant and nearly free, so any gain we claim must exceed it. We run it on the same shape backbone and the same anchor subsamples.
The strongest anchored competitors are ScaleMapLearner refinement from the same paper and LiDARTouch-style dense fusion. On KITTI, LiDARTouch reports RMSE 2.504 m with 4 beams, against 2.112 m for a 64-beam supervised model, and a catastrophic distance rate of 0.0%. For the ScaleMapLearner refinement, the file gives only a relative figure: up to 30% iRMSE reduction over global alignment and over 50% lower iRMSE than KBNet with 150 points. We found no reported absolute VOID number on Lacuna. We retrain both competitors on our backbone, with identical training data, anchor sampling, and tuning budget measured in GPU-hours. We also report LiDARTouch's published number for reference. On in-range KITTI, beating these means matching or improving RMSE. On out-of-range evaluation, it means a clear margin.
For the zero-anchor regime, the strongest methods are Metric3D v2, which reports zero-shot AbsRel of 0.044 on KITTI and 0.047 on NYUv2 with ViT-Large, and ScaleDepth, which reports ARel 0.048 on KITTI. We do not expect our top-1 prediction to beat Metric3D v2 trained at its data scale. Instead we compare at matched data: top-1 against ScaleDepth-style single-scale regression on our backbone, and best-of-K to measure how much error is due to scale mode collapse.
Two ablations isolate the key ingredients. The first keeps the identical architecture but removes exact canonicalization, replacing it with log-uniform scale jitter of anchors and targets over the training range. This tests whether the guarantee itself matters. The second sets $K = 1$ to isolate the symmetry-breaking set.
Why not just…?
Why not just use scale-jitter augmentation instead of exact equivariance? This is the real threat. The human-pose study shows augmentation winning over architectural equivariance for rotations. The difference here is that rotations form a bounded group, which augmentation can cover completely, while scale is unbounded. Jitter can only cover a chosen range. The augmented ablation is designed to settle this. We train it with jitter covering the training depth range and evaluate it beyond that range. If it matches the equivariant model even far out of range, we will report that the guarantee is unnecessary and keep only the hypothesis-set contribution.
Why not just align relative depth to the anchors by least squares? It is already exactly equivariant and is our naive baseline. Its weaknesses are a single global degree of freedom, which cannot correct local distortion, and sensitivity to outlier anchors. Prior work reports learned refinement cutting iRMSE by up to 30% over it. Our model must beat it at 1 to 4 anchors and under injected anchor outliers. If it does not, the learned module is not earning its cost.
Why not just regress one scale from semantics or appearance, as ScaleDepth or Metric3D v2 do? A single regressed scale must average across plausible scales when the image is ambiguous. ScaleDepth's own authors note failures on out-of-vocabulary scenes. The $K = 1$ ablation and the best-of-K gap measure how much accuracy is lost to this averaging. If the gap is negligible on real benchmarks, the multi-hypothesis head is not worth keeping, and we will say so.
Proposed Experiments
The decisive experiment is range extrapolation. We train on indoor data only, NYUv2 and VOID, where depths stay within roughly ten meters. We then test on KITTI with 4-beam subsampled LiDAR anchors and with 1-point anchors, reporting AbsRel, RMSE, and catastrophic distance rate. A dense-fusion network or the augmented ablation should degrade sharply here. Our model should stay close to its in-range accuracy. We run the reverse direction, KITTI to NYUv2, as well.
Second, we sweep anchor count from 150 down to 0 on VOID and KITTI, and also corrupt anchors with Gaussian log-noise and a 20% outlier fraction. The resulting curves locate where each method breaks.
Third, in the zero-anchor regime we report top-1 error, best-of-K error, and selector calibration on NYUv2 and KITTI. We also run a synthetic check on TartanAir: render scenes, uniformly rescale them, and confirm that the hypothesis set places mass on the multiple scales consistent with the training distribution rather than on their average.
The proposal succeeds if exact equivariance yields a clear out-of-range advantage over the augmented ablation, and if it holds or improves in-range accuracy relative to retrained LiDARTouch and ScaleMapLearner baselines on matched compute. It fails, informatively, if augmentation closes the gap.
References
- Equivariant Symmetry Breaking Sets
- ScaleDepth: Decomposing Metric Depth Estimation into Scale Prediction and Relative Depth Estimation
- Monocular Visual-Inertial Depth Estimation
- LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- Towards Zero-Shot Scale-Aware Monocular Depth Estimation
- On the Role of Rotation Equivariance in Monocular 3D Human Pose Estimation
- SE(3) Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation