Models trained on simulator images are cheap to label and plentiful, but when they are run on real camera images they make large errors. Worse, they give no warning about which outputs are wrong. Regression tasks make this harder. In monocular depth estimation (predicting distance at every pixel from one image) or object pose estimation, the network outputs an unbounded number with no built-in measure of how sure it is. Self-training, where a model labels unlabeled real images and then trains on its own guesses, therefore has no reliable way to tell good guesses from bad ones. Bad guesses then reinforce themselves.
We propose to replace "how uncertain is this number?" with a question that has a checkable answer: "what is the probability that this prediction lands within the tolerance the task's own accuracy metric uses?" For depth, the tolerance is the standard ratio test, where predicted and true depth must differ by less than a factor of 1.25. For pose, it is the threshold behind the reported recall. A small head predicts this probability. We train it on heavily augmented synthetic images, where the true error is known, and explicitly calibrate it there. Because the event is binary, the classification calibration machinery that is known to transfer across the sim-to-real gap becomes usable for regression. The calibrated probability then serves directly as the per-pixel weight in single-image self-training on real data. The method needs no stereo pairs, video, or CAD-based refinement.
Core Claim
AUGCAL showed that sim-to-real adaptation makes classifiers confidently wrong. It also showed that a calibration loss applied to augmented synthetic images reduces target miscalibration, with ECE on GTAV→Cityscapes falling from 5.34 to 3.43. However, it is defined only for class probabilities. Regression has no analogue of its confidence-versus-accuracy gap. Our claim is that defining confidence as a tolerance event closes this mismatch.
The head outputs $p(x,i)\approx \Pr[\,\max(\hat d_i/d_i,\, d_i/\hat d_i) < 1.25 \mid x\,]$. Here $\hat d_i$ and $d_i$ are the predicted and true depth at pixel $i$ of image $x$. We train it with binary cross-entropy plus AUGCAL's confidence-minus-accuracy penalty on augmented synthetic images. Gradients are stopped into the depth network, so the head learns to predict the errors that the deployed predictor actually makes under simulated shift. It does not learn to make those errors smaller.
The testable prediction is twofold. First, calibration of $p$ measured on augmented synthetic data should predict calibration on real KITTI and NYUv2, which we can check because both have ground-truth depth that is withheld during training. Second, $p$-weighted self-training should beat both unweighted consistency training and variance-based weighting.
This differs from existing work in specific ways. MAST also restores a softmax by discretizing pose into bins and thresholding on the bin classifier. It leaves the fine residual error unguarded and needs bin hyperparameters. Our event is defined on the final continuous error, and the output space is never binned. 3D-PL selects pseudo-labels by checking whether a prediction stays stable under real-to-synthetic style transfer. Stability is not correctness: a shortcut-driven error can be perfectly stable. Our confidence is trained against true error.
Research Program
The base predictor is a depth network pretrained on Virtual KITTI 2 (Virtual KITTI 2) or SceneNet. The confidence head is a lightweight decoder branch on the shared features.
The augmentation family is the main design lever. If the augmentations do not cover the real shift, calibration will not transfer. We will use AUGCAL's appearance augmentations, which are PASTA-style and RandAugment-style. We add sensor-motivated corruptions: noise, blur, clipping of the brightness range, and lighting changes. These target the rendering and sensor sources of the gap described by Schels et al. and Rafferty et al.. We also hold out augmentation subsets and ask which ones carry calibration over to real data. This analysis is useful even if the self-training gains turn out small.
Adaptation is a teacher–student loop on single real frames. The teacher is a slowly updated copy of the student. Its predictions supervise the student through a loss weighted per pixel by $p$. We keep the source supervised loss, and $p$ keeps being refreshed on augmented synthetic data so it stays calibrated as the features drift. We deliberately avoid matching real pseudo-label distributions to the synthetic label distribution. RankUp states that its distribution alignment step assumes shared label marginals, and that assumption fails between synthetic and real driving scenes.
A second testbed checks that the idea is not depth-specific. On LineMOD and Occluded LineMOD, with synthetic PBR training images, the tolerance event is the pose-correctness criterion behind recall. That lets us test against MAST in its own setting.
Baselines
Source-only reference. The naive reference point is the source-only network: trained on augmented synthetic data and applied to real images without adaptation. It shows how much of any gain comes from augmentation alone.
SemiSupMDE (strongest existing single-frame competitor for depth). SemiSupMDE enforces agreement across one original and two strongly augmented real views without any confidence weighting. It reports AbsRel 0.161 on vKITTI→KITTI and RMSE 1.055 on SceneNet→NYUv2. It is the strongest competitor because it uses only single real frames, as we do, and it beat 3D-PL, which reported AbsRel 0.162.
To make the comparison fair, we will rerun SemiSupMDE with our backbone, our source data, and our augmentation family. We will match training steps and tuning budget, and report both reproduced and published numbers. Beating it means lower AbsRel than our reproduction on KITTI and lower RMSE on NYUv2, consistent over three seeds. Matching it at substantially less than its roughly 35GB memory footprint would count as a partial win. Our method needs one target stream plus one augmented view, against SemiSupMDE's five images.
Back2Color (cited for context, not compared). Back2Color reports AbsRel 0.087 on KITTI. It uses stereo and temporal cues during adaptation, so it is not a fair single-frame comparison. We will cite it as an upper reference that uses more information.
MAST (strongest existing method for pose). On pose, the strongest baseline is MAST, which reports 87.2% average recall on LineMOD and 61.4% on Occluded LineMOD. We will replace its bin-softmax confidence with our tolerance head while keeping its manifold regularizer and its self-paced schedule. Any change in recall can then be attributed to the confidence signal.
Ablations isolating the key ingredient. The first ablation is a Gaussian variance head trained by negative log-likelihood on the same augmented synthetic data, with $p$ computed analytically from the predicted variance. This is in the spirit of heteroscedastic pseudo-labels and UCVME. The second ablation is our head with the calibration penalty removed. The third trains our head on clean rather than augmented synthetic images.
Why not just…?
Why not just predict a Gaussian variance and convert it to a tolerance probability? This is the most serious threat. Negative log-likelihood is a proper scoring rule, so on augmented data it may calibrate itself without any extra machinery. Our reason to expect a gap is that a Gaussian assumes symmetric, unimodal errors. Sim-to-real depth errors are often systematic: whole surfaces placed at the wrong scale, or textureless regions that are confidently misplaced, which is the "confidently wrong" behavior documented in Synthetic-to-Real Pose Estimation with Geometric Reconstruction. The variance-head ablation decides this directly. If its real-domain calibration and downstream AbsRel match ours, the contribution narrows to the empirical finding that augmented-synthetic calibration transfers for regression, and we will report it that way.
Why not just apply post-hoc variance scaling on a synthetic validation set? AUGCAL found that post-hoc temperature scaling worsened overconfidence under sim-to-real shift. A single global scale also cannot reorder pixels. Pseudo-label weighting depends on that ordering, not just on the average uncertainty level. We will include a fitted global scale on top of the variance head and report its real-domain calibration.
Why not just use agreement across augmentations as the confidence? This is what 3D-PL and SemiSupMDE implicitly do, and it costs no training. Its weakness is that errors driven by simulator shortcuts can be consistent across augmentations. We will measure on real KITTI how well disagreement separates correct from incorrect pixels, compared with our $p$, using area under the ROC curve for the within-tolerance event.
Why not just use an ensemble or MC dropout? Running several dropout-perturbed passes added about 2.3× compute in UCVME. Ensemble members trained on the same synthetic data also share the same synthetic biases. We will include a five-member MC-dropout baseline in the discrimination analysis to measure this directly.
Proposed Experiments
Experiment 1 (decisive test of the core claim). The first experiment measures real-domain calibration and discrimination of the within-tolerance event on KITTI (Eigen split) and NYUv2, using ground truth only for evaluation. It compares our head, the variance head, augmentation disagreement, and MC dropout, using expected calibration error on $p$ and area under the ROC curve. The claim fails if augmented-synthetic calibration does not reduce real calibration error relative to clean-synthetic training.
Experiment 2 (adaptation accuracy). The second experiment is $p$-weighted self-training on vKITTI2→KITTI and SceneNet→NYUv2. We report AbsRel, Sq Rel, RMSE, and the 1.25-threshold accuracy against the reproduced SemiSupMDE. We also report peak GPU memory.
Experiment 3 (generalization beyond depth). The third experiment is the pose transfer on LineMOD and Occluded LineMOD against MAST.
Experiment 4 (which augmentations matter). The fourth experiment holds out augmentation subsets and correlates each subset's synthetic calibration with real calibration. This tells practitioners which simulated corruptions make confidence trustworthy.
References
- AUGCAL: Improving Sim2Real Adaptation by Uncertainty Calibration on Augmented Synthetic Images
- Manifold-Aware Self-Training for Unsupervised Domain Adaptation on Regressing 6D Object Pose
- 3D-PL: Domain Adaptive Depth Estimation with 3D-Aware Pseudo-Labeling
- Virtual KITTI 2
- Learning an object class representation on a continuous viewsphere
- Sequential feature selection for power system event classification utilizing wide-area PMU data
- RankUp: Boosting Semi-Supervised Regression with an Auxiliary Ranking Classifier
- Consistency Regularisation for Unsupervised Domain Adaptation in Monocular Depth Estimation
- Back2Color: Domain-Adaptive Synthetic-to-Real Monocular Depth Estimation for Dynamic Traffic Scenes
- Semi-Supervised Regression with Heteroscedastic Pseudo-Labels
- Semi-Supervised Deep Regression with Uncertainty Consistency and Variational Model Ensembling via Bayesian Neural Networks
- Synthetic-to-Real Pose Estimation with Geometric Reconstruction