Simulators hand us something real data never does: a perfect depth map for every pixel. A common way to use it when teaching a segmentation model to work on real street images is to train a "teacher" that sees both color and depth on synthetic scenes. That teacher then labels unlabeled real photos, and an RGB-only "student" learns from those labels, a setup called cross-modal knowledge distillation. The catch is that real photos come without depth, so the teacher is fed depth estimated from a single image. Where that estimate is wrong (glass, sky, shadows, thin poles), the teacher confidently mislabels pixels, and the student learns those mistakes. We propose a direct fix. Estimate, pixel by pixel, how wrong the depth input is. Calibrate that estimate on real images using geometry-only supervision that is cheap to obtain, such as stereo disparity. Then let the depth-aware teacher supervise only the pixels where its depth input can be trusted, and fall back to an RGB-only teacher everywhere else.
The bet is that the error of the guiding modality predicts pseudo-label errors that output confidence misses. A depth-aware teacher can be very confident precisely because a hallucinated depth edge gave it a clean boundary. If this holds, gating on input reliability should beat gating on output confidence, and the student should degrade gracefully, instead of collapsing, when depth estimation fails. If it does not hold, existing confidence-based filters already suffice. We design the first experiment to tell these apart cheaply before building anything else.
Core Claim
In synthetic-to-real adaptation with privileged depth, most of the harmful noise in cross-modal pseudo-labels comes from errors in the estimated depth fed to the teacher on real images, not from the teacher's semantic uncertainty. This noise can be detected without semantic labels. A depth-error predictor is trained on synthetic data, where true depth is known, and is then recalibrated on real images against stereo-derived depth. The calibration therefore happens in the target domain, which sidesteps the broken exchangeability that makes source-calibrated uncertainty untrustworthy under shift.
Two pieces of prior work motivate this claim. CrossMatch adapts source-free from GTA5 or SYNTHIA to Cityscapes, using an auxiliary RGB-D stream to supervise the RGB stream through cross-modal consistency and prototype-based denoising. Its own stated limitation is sensitivity to depth-estimation failures, and nothing in the method models where depth is wrong. Cross-Domain 3D Hand Pose Estimation with Dual Modalities predicts pseudo-depth on real RGB and feeds it back to refine pose, with no uncertainty on that depth. This is the cross-modal version of the error-feedback loop described in Tri-net, where unverified pseudo-labels compound their own errors over training.
Two other papers supply the uncertainty machinery and also mark how we differ. AdaConG uses split conformal prediction, a distribution-free method that turns a calibration set into prediction sets with guaranteed coverage, and downweights guidance when the teacher's prediction set is large. It looks only at the teacher's output, and only in single-modality settings. UncerDA fits class-wise two-component Gaussian mixtures to prediction entropy to separate certain from uncertain pseudo-labels without fixed thresholds. It is purely RGB. We instead quantify uncertainty in the teacher's privileged input and ask whether that adds information beyond these output-side signals.
Research Program
The system builds on CrossMatch's source-free pipeline, so the only change is the gate. There are two teachers. One sees real RGB plus estimated depth; the other sees RGB only. A small depth-error head is attached to the depth-aware teacher and trained on synthetic source data to regress the per-pixel error between monocular-estimated depth and rendered ground-truth depth. On the unlabeled Cityscapes training split, we compute a stereo-versus-monocular disparity discrepancy for every image. On a held-out subset of those images, we fit a conformal threshold $q$ so that, among pixels whose predicted error falls below $q$, the fraction whose stereo-measured depth error exceeds a tolerance $\tau$ is at most $\alpha$. No semantic labels are used at any point in training or calibration.
The student, with prediction $f_\theta$ at pixel $p$, is trained with
$$\mathcal{L}(p) = g_p \, \mathrm{CE}\big(f_\theta(p), \hat y^{\text{RGBD}}_p\big) + (1-g_p)\, \mathrm{CE}\big(f_\theta(p), \hat y^{\text{RGB}}_p\big),$$
Here $\hat y^{\text{RGBD}}_p$ and $\hat y^{\text{RGB}}_p$ are the two teachers' pseudo-labels, and $g_p$ is 1 when the predicted depth error $\hat e_p \le q$ and 0 otherwise. We also test a soft sigmoid version of the gate. Any existing output-side filtering, whether CrossMatch's prototypes or a confidence or GMM filter, still applies on top of this. The gate decides which teacher to listen to, not whether to listen at all.
This gate can be read as a local, testable stand-in for the condition in Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning. That paper argues that cross-modal distillation helps only when the teacher's representation carries enough information beyond what the student already has. Its proof is limited to Gaussian, square-loss settings, so we use it as motivation and not as a guarantee. A pixel with bad depth is one where the depth-aware teacher has nothing correct to add.
A second regime tests the case without stereo: RHD (synthetic RGB-D hands) to FreiHAND and STB (real RGB). Here the error head must be trusted as trained on synthetic data, with only source-side calibration. That is exactly the setting where we expect calibration to weaken, and we want to measure how much.
Baselines
The naive reference point is the RGB-only teacher's pseudo-labels with no depth at all. This is the source-free model self-trained on its own predictions, and it tells us whether privileged depth helps on Cityscapes in the first place.
The strongest existing method is CrossMatch. It reports 57.7% mIoU on GTA5→Cityscapes and 57.5% on SYNTHIA→Cityscapes in the source-free setting. We will reimplement it with the same backbone, monocular depth estimator, schedule, and tuning budget, confirm we land within about one point of the reported numbers, and then add only the gate. A clear win is a gain of at least one mIoU point over our reproduced CrossMatch across three seeds, with gains concentrated in classes and regions where depth is known to fail.
We also run CrossMatch fed stereo depth instead of monocular depth. This checks whether the gate does anything beyond simply having better depth.
The output-side competitors are UncerDA, which reports 52.6% mIoU on GTA5→Cityscapes in its non-source-free setting and is cited for reference only, and an AdaConG-style conformal gate on the depth-aware teacher's output prediction sets. We run both inside the same CrossMatch pipeline at equal tuning budget, in place of our input-error gate.
DADA reports 42.6% mIoU on SYNTHIA→Cityscapes, against 40.8% for AdvEnt. It represents the depth-as-auxiliary-task alternative and is run under its published settings.
Three ablations isolate the key ingredient. The first gates on the stereo error itself, as an oracle upper bound. The second uses a source-only calibrated threshold, which tests whether target-side calibration matters. The third uses a random gate matched in the fraction of pixels gated.
For hands, the reference is the dual-modality method's 15.61mm FreiHAND end-point error and its 16.0% relative 2D improvement on STB over SemiHand. We compare against our reproduction of it with and without the gate.
Why not just…?
Why not just filter the depth-aware teacher's pseudo-labels by confidence or entropy? This is the real threat. If high depth error always shows up as high teacher entropy, then UncerDA-style mixtures or AdaConG-style prediction sets already catch these pixels, and our gate is redundant. The first diagnostic below is built to settle this before any training. If depth error adds negligible ability to predict pseudo-label mistakes once entropy is known, we abandon the gate and report that negative result.
Why not just use stereo depth directly instead of a monocular estimate? Stereo is also wrong in specific places: occlusion boundaries, sky, textureless and reflective surfaces. It is also unavailable in many deployments, including our hand-pose regime. The "CrossMatch with stereo depth" baseline measures how much better depth alone buys. If it matches the gated method, the honest conclusion is "use stereo when you have it," and the gate's value shrinks to stereo-free settings.
Why not just predict depth as an auxiliary task, as DADA does? Multi-task training makes the backbone share capacity between depth and semantics, but it does not stop wrong depth from shaping pseudo-labels. CrossMatch reports beating auxiliary-depth approaches on GTA5→Cityscapes. We include DADA so the comparison is run under one protocol rather than taken on trust.
Why not drop depth and self-train RGB only? Then the gap DADA and CrossMatch exploit disappears, and geometry stops anchoring near-object classes; DADA attributes its car and bus gains to depth. The naive baseline quantifies how much depth is worth in our reproduction. Our method should never fall below it, because the gate defaults to the RGB teacher.
Proposed Experiments and Decisive Checks
The first, kill-or-continue check uses Cityscapes validation labels for analysis only. We take the reproduced CrossMatch depth-aware teacher and compute how well its pixel errors are predicted by entropy alone, by stereo-measured depth error alone, and by both together, measured by area under the ROC curve. We proceed only if adding depth error raises it meaningfully. We would set a pre-registered bar of about three points, chosen before looking.
The main experiment compares mIoU on GTA5→Cityscapes and SYNTHIA→Cityscapes against all baselines above, with per-class breakdowns. We expect the largest gains on poles, signs, glass-fronted buildings, and distant objects.
A stress test corrupts the monocular depth on real images with blur, global scale bias, and dropped regions. Ungated CrossMatch should decline toward or below the RGB-only reference, while the gated student should stay at or above it. This is the "graceful fallback" claim, and a failure here falsifies it directly.
The calibration check reports the realized violation rate of the depth-error guarantee on Cityscapes validation. We do this for both target-calibrated and source-calibrated thresholds, which quantifies how badly shift breaks source-side calibration.
Finally, the hand-pose transfer from RHD to FreiHAND and STB tests the method without stereo. A modest gain, or none, there would mark the honest boundary of the approach: target-side geometric calibration is what makes it work.
References
- CrossMatch: Source-Free Domain Adaptive Semantic Segmentation via Cross-Modal Consistency Training
- Cross-Domain 3D Hand Pose Estimation with Dual Modalities
- Tri-net for Semi-Supervised Deep Learning
- Adaptive Conformal Guidance for Learning under Uncertainty
- Uncertainty-aware Pseudo Label Refinery for Domain Adaptive Semantic Segmentation
- Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning
- DADA: Depth-aware Domain Adaptation in Semantic Segmentation