Which Part of the Self-Supervised Depth Gap Can a Curriculum Close?

Self-supervised monocular depth networks learn depth from ordinary video. They do this without ground-truth labels: they predict depth and camera motion, warp a neighbouring frame into the current view, and penalize the pixel-colour mismatch, which is called the photometric loss. These models still trail models trained on LiDAR depth, and the gap is worst at object edges and thin structures. Two quite different things could cause this. In some places the photometric loss genuinely prefers a wrong depth, for example on reflective surfaces, at occlusions, or at anti-aliased boundaries. In other places the loss would prefer the true depth, but training never finds it. A curriculum is a schedule that presents easy training signal before hard. It can only help with the second kind of failure. Yet no one has measured how the gap splits between the two.

This proposal does that measurement and then uses it. For every pixel with a LiDAR reading, we ask whether the true depth would score a lower photometric loss than the depth the network learned. If it would, the network is under-optimized there. If it would not, the loss itself is wrong there. We then test one specific curriculum aimed at the under-optimized part: the loss is first blurred and the ambiguous edge pixels are masked, and both are relaxed over training. The prediction is concrete. Permanently suppressing high-frequency supervision, as current methods do, trades the optimization problem at edges for a change in the objective. Relaxing that suppression late in training should recover edge accuracy that static masking gives up. The measurement tells us beforehand how much accuracy is available to recover.

Core Claim

The gap between self-supervised and supervised monocular depth has two parts.

  • Objective gap. These are pixels where the ground-truth depth has higher photometric error than the learned depth. No training schedule can fix these, because the loss itself points the wrong way.
  • Optimization gap. These are pixels where the ground-truth depth would score lower error but the network did not reach it. A curriculum can only act on this part.

We claim two things. First, a large share of the edge-region gap is optimization gap. Second, a frequency curriculum can close it, where static fixes cannot. The curriculum starts with heavy smoothing of the loss map and masking of boundary pixels, then anneals both to zero.

Because each pixel's reprojection depends only on that pixel's depth and the camera pose, the split is clean. For a target pixel $p$ with LiDAR depth $d^*_p$, define the per-pixel gap indicator as

$$ g_p = \mathrm{PE}_p(d^*_p, T^*) - \mathrm{PE}_p(\hat d_p, \hat T), $$

where:

  • $\mathrm{PE}_p$ is the minimum-over-source-frames photometric error at $p$;
  • $\hat d_p$ and $\hat T$ are the network's median-scaled depth and its pose;
  • $T^*$ is a pose re-fitted, with only six degrees of freedom, to minimize photometric error given the ground-truth depths. This keeps the comparison from being biased toward the network's own pose.

When $g_p < 0$ and the depth error is large, $p$ belongs to the optimization gap. When $g_p \ge 0$ and the depth error is large, $p$ belongs to the objective gap. We aggregate these indicators by region type: edge band, smooth surface, thin structure, and pixels the auto-mask flags as stationary.

Relation to Prior Work

Work that motivates the split. Several papers find the gap and blame the proxy signal.

Neither paper separates errors the loss endorses from errors the loss rejects but training fails to escape. That separation is our first contribution.

Work that supplies the static fixes we compete with. - Digging Into Self-Supervised Monocular Depth Estimation (Monodepth2) introduces three changes: taking the minimum reprojection error across source frames, a binary auto-mask for stationary pixels, and full-resolution multi-scale losses. - Frequency-Aware Self-Supervised Monocular Depth Estimation argues that high-frequency regions give huge photometric penalties for tiny depth errors. Its two fixes are permanent: it masks anti-aliased boundary pixels (Ambiguity-Masking) and low-pass filters the loss map (Auto-Blur).

Our hypothesis is that those permanent fixes help early optimization but change the objective at exactly the pixels where supervised models are sharpest. That makes them a target for annealing rather than removal.

Work that supplies the curriculum mechanism. - Curriculum DeepSDF anneals an error tolerance toward zero. This lets 3D shape networks fit coarse geometry before thin parts. - FastDINOv2: Frequency Based Curriculum Learning Improves Robustness and Training Speed paces input frequency during self-supervised pre-training.

We borrow the annealing idea but apply it to the frequency content of a self-supervised geometric loss, not to known 3D distances or input images.

The closest curriculum work in self-supervised depth. WeatherDepth: Curriculum Contrastive Learning for Self-Supervised Depth Estimation under Adverse Weather Conditions orders training by synthetic weather severity. It switches stages when a global loss plateaus. Its difficulty axis is external image corruption, and it never asks whether the remaining error is reachable. Ours is the internal frequency structure of the loss in clear weather, with a per-pixel diagnostic deciding where the curriculum should act.

Work that addresses the gap in other ways. - Calibrating Self-supervised Monocular Depth Estimation handles scale ambiguity. We sidestep scale by median scaling in both the evaluation and the diagnostic. - Adaptive Co-teaching for Unsupervised Monocular Depth Estimation trains a multi-decoder ensemble to handle unreliable regions. Our approach adds no parameters.

Theory that connects the two directions. A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula proves that easy-to-hard schedules beat uniform mixtures when three conditions hold:

  • the initial model can already solve the first stage;
  • stage gaps are moderate;
  • the sample budget is sufficient.

It also assumes a perfect verifier. Our objective gap is, in effect, a direct measurement of how imperfect the photometric "verifier" is, pixel by pixel. That is the regime the theory leaves open.

Work that constrains the design. Self-Paced Learning for Latent Variable Models selects low-loss samples first and anneals a threshold. The Frequency-Aware analysis implies that loss magnitude is a poor difficulty signal here, because nearly correct edge pixels carry large loss. A naive self-paced rule would therefore postpone exactly the pixels we want. We test this as a baseline rather than assume it.

Research Program

Stage one: the diagnostic. We train four models on the KITTI Eigen split, all with Monodepth2's ResNet-18 at matched resolution and training length:

  1. plain Monodepth2;
  2. Monodepth2 with static Ambiguity-Masking and Auto-Blur;
  3. the same network trained with direct LiDAR supervision, as a supervised reference;
  4. Monodepth2 with a larger backbone, to see how capacity alone shifts the split.

On the test LiDAR points we compute $g_p$ and depth error. We report what fraction of the total error above the supervised model's error falls into the objective gap versus the optimization gap, stratified by region type. Occluded pixels are treated separately; we identify them as pixels where no source frame gives a valid reprojection.

Stage two: the curriculum. We train Monodepth2 with Auto-Blur and Ambiguity-Masking whose strengths decay on a schedule:

  • the blur width $\sigma(t)$ falls from the static method's value to zero;
  • the masked boundary band shrinks to nothing.

The schedule family covers linear, cosine, and a hard two-step switch at fraction $\tau$ of training. A small grid over the decay start and end points is tuned on a held-out part of the KITTI training split, using photometric loss and no ground truth. This keeps the method label-free.

Stage three: the theory link. We relate the schedule results to the curriculum conditions from the task-centric theory above. Specifically, we test whether schedules that drop the blur too early reproduce the failure the theory predicts for overly large stage gaps.

Baselines

The naive reference: plain Monodepth2. This is Monodepth2 trained with its standard losses, retrained in our codebase with an identical schedule and data. Digging Into Self-Supervised Monocular Depth Estimation reports AbsRel 0.115 on the KITTI Eigen split for monocular training. Our reproduction must land near this number before any comparison is trusted.

The strongest direct competitor: static Frequency-Aware Monodepth2. This is Monodepth2 with Ambiguity-Masking and Auto-Blur applied for the whole run, as in Frequency-Aware Self-Supervised Monocular Depth Estimation. That paper reports AbsRel 0.112, SqRel 0.834 and RMSE 4.746 for this combination, against its own Monodepth2 baseline of 0.183. The file does not explain why that baseline sits far from Monodepth2's own 0.115, so the gain is probably reported under a different setting. We therefore rerun both methods ourselves under one protocol. The annealed curriculum receives the same tuning budget as this baseline's fixed hyperparameters, plus the small schedule grid, which we report.

Beating the competitor. The curriculum beats the static method if both of the following hold over three seeds:

  • it lowers AbsRel and RMSE in the edge and thin-structure bands;
  • it does not raise global AbsRel.

A strong reference that is not a like-for-like competitor: Adaptive Co-teaching. Adaptive Co-teaching for Unsupervised Monocular Depth Estimation reports AbsRel 0.104 at high resolution with ResNet-18. It uses a multi-decoder teacher ensemble. We list it as context rather than as a matched competitor, and we test whether our schedule stacks on top of its student training.

The supervised reference. This is the same network trained on LiDAR depth. It defines the gap we decompose. No reported result was found on Lacuna for a matched supervised ResNet-18 Monodepth2-style model on the KITTI Eigen split, so we train it ourselves.

Ablations that isolate the key ingredient. Two ablations separate the schedule from its contents:

  • Anneal blur only. This tests whether the gain comes from restoring high-frequency supervision through the blur.
  • Anneal mask only. This tests whether the gain comes from re-admitting the boundary pixels.

A random-order variant, which starts unblurred and adds blur late, tests whether the ordering matters rather than mere variation in the loss.

Why not just…?

Why not just fine-tune without the masks for the last few epochs? This is the most serious threat. A two-step switch is a legitimate member of our schedule family, so we include it explicitly. If it matches smooth annealing, we report that the useful ingredient is "suppress early, release late" and that the schedule's shape is irrelevant. Our core claims would survive, because they concern whether release helps at all and how much of the gap is reachable. That is a cheaper result, and we would recommend it.

Why not just use a bigger backbone or higher resolution? Bridge the Gap between Supervised and Unsupervised Learning for Fine-Grained Classification reports a 24.1% boost on CUB from a backbone swap alone. Capacity clearly matters. Against this, WeatherDepth found that a MonoViT backbone still left AbsRel at 0.120 on WeatherKITTI, until its curriculum brought it to 0.103. The diagnostic settles this case directly. If the larger backbone shrinks the edge-band optimization gap as much as the curriculum does, the curriculum is unnecessary at that scale. We also run the curriculum on the larger backbone to check whether the two effects add.

Why not just weight pixels by their loss, self-paced style? We expect this to fail, because large loss at edges signals sharp texture, not a wrong answer. That is precisely the point of Frequency-Aware Self-Supervised Monocular Depth Estimation. We run a per-pixel version of the annealed-threshold rule from Self-Paced Learning for Latent Variable Models at a matched budget. If it matches our curriculum, our explanation is wrong, and we will say so.

Proposed Experiments and Decisive Checks

All models are trained on the KITTI Eigen split and evaluated with AbsRel, SqRel, RMSE, log RMSE and the three δ thresholds, after median scaling. We add error restricted to three bands:

  • an edge band around LiDAR depth discontinuities;
  • a band around high-gradient image regions;
  • thin structures.

Zero-shot transfer to Cityscapes tests whether recovered edge accuracy survives a change of dataset.

Decisive check one. The diagnostic must show a non-trivial optimization-gap share in the edge band. If nearly all edge error is objective gap, curriculum learning cannot close this gap for photometric losses, and the paper's main result becomes that negative finding.

Decisive check two. Annealing must move edge-band pixels from the optimization gap toward the supervised model's error, compared with the static Frequency-Aware method. At the same time, it must not grow the objective gap in smooth regions.

Decisive check three. The two-step switch and the loss-ranked self-paced baseline must be separated from smooth annealing by more than seed variance; otherwise we report the simpler method.