Models trained on rendered or simulated images are adapted to real cameras with a standard trick: push the model to make confident predictions on unlabeled real images. The hope is that decision boundaries then move into empty regions between real-data clusters. The trick raises accuracy, but it is applied to every real image, including ones the simulator never could have produced, such as unmodeled lighting, clutter, or objects outside the label set. On those images the model is forced to be confidently wrong. Function-space regularization offers a principled alternative. It places a prior directly on the model's predictions at chosen "context points," for example "be uncertain here," instead of on its weights. Its known weak spot is that nobody has a good way to choose those points.
This note proposes to let the simulator choose them. We score each unlabeled real image by how well a heavily randomized version of the simulator covers it. Where coverage is high, the function-space prior is centered on the synthetic-trained model's predictions, and confidence-seeking adaptation is allowed. Where coverage is low, the prior reverts to a wide, high-entropy distribution, and the model is required to stay uncertain. The claim is that this keeps most of the accuracy gains of confidence-seeking adaptation while greatly improving calibration and the rejection of real content the simulator does not model. Current sim-to-real methods lack that second property, and it matters most for deployment.
Core Claim and Position Relative to Prior Work
Three sets of papers shape this proposal, each in a different way.
The first set motivates the core tension. A DIRT-T Approach to Unsupervised Domain Adaptation minimizes prediction entropy on unlabeled target images. It also penalizes output changes under small adversarial input perturbations. It then refines the model with a teacher–student KL term. It reaches 76.5% on MNIST→SVHN and 99.6% on Synthetic Signs→GTSRB. However, it keeps no uncertainty over functions and pushes every target input toward low entropy. The function-space literature shows why that is risky. Noise Contrastive Priors for Functional Uncertainty demonstrates that weight-space priors leave predictions unconstrained, and often overconfident, off the training data. It fixes this by penalizing the model toward a wide output distribution on noise-perturbed inputs.
The second set supplies the method. Tractable Function-Space Variational Inference in Bayesian Neural Networks makes a function-space KL term tractable by linearizing the network around its mean weights. Function-Space Regularization in Neural Networks: A Probabilistic Perspective (FS-EB) penalizes a Mahalanobis distance to a reference prior on context points. It reports AUROC 99.4% on CIFAR-10 vs. SVHN, at roughly 3x compute cost from Jacobians. All of these papers choose context points by heuristics: Gaussian noise, monochrome images, or an unrelated real dataset. Drug Discovery under Covariate Shift with Domain-Informed Prior Distributions over Functions is the closest in spirit. It anchors a function-space prior on about 2 million unlabeled molecules and improves AUC-ROC under scaffold shift, from 0.589 for deep ensembles to 0.606. Its context points share one uniform prior, with no notion of which points a generator could explain.
The third set constrains the design. Bridging the Sim-to-Real Gap with Bayesian Inference (SIM-FSVGD) is the only work in the file that uses a simulator as a functional prior. It does this for low-dimensional dynamics, 12-dimensional state vectors, using a Gaussian score approximation and querying the simulator in the loop. That approach does not transfer to images, because one cannot sample a renderer's distribution over classifier outputs per pixel. We therefore use the simulator differently. It does not supply prior function samples. It decides where the prior is informative and where it is wide.
DeceptionNet: Network-Driven Domain Randomization constrains the design in another way. It shows that adversarially randomized renders generalize to unseen real backgrounds: 89.4% on MNIST-COCO, where PixelDA fell to 72.5%. That result is why we build coverage from a randomized simulator rather than a fixed one.
What is new is the per-point, simulator-derived choice between two priors inside one function-space objective. A narrow prior centered on the teacher applies where the randomized simulator covers the real image, and a wide, uniform-leaning prior applies where it does not. The same coverage score gates the entropy term.
Research Program
We first train a source model on synthetic data with DeceptionNet-style randomization: background replacement, elastic distortion, pixel noise, and lighting changes. From this frozen model we extract features for a large pool of randomized renders. For each unlabeled real image $x$, we compute a coverage score $c(x) \in [0,1]$ from a density estimate relative to that pool. Our first choice is a k-nearest-neighbour distance, which we calibrate on held-out randomized renders so that $c$ is about 1 on simulator-producible inputs. Coverage is computed from frozen features, not from the model being adapted. This avoids the circularity of a model certifying its own confidence. The features are refreshed at fixed intervals.
The adapted model $f$ is trained with the following objective:
$$ \mathcal{L} = \mathcal{L}_{\text{src}} + \lambda_{e}\sum_{x\in T} c(x)\,H\big(f(x)\big) + \lambda_{f}\sum_{x\in T}\mathrm{KL}\big(q(f(x))\,\|\,p_{c(x)}(f(x))\big). $$
The symbols are defined as follows. $\mathcal{L}_{\text{src}}$ is the labeled cross-entropy on randomized synthetic data. $T$ is a minibatch of real context points. $H$ is predictive entropy. $q(f(x))$ is the model's linearized Gaussian distribution over logits, following the FSVI approach. $p_{c}$ is a Gaussian prior over logits. That prior has mean $c\,m(x)$ and variance $\sigma_0^2 + (1-c)\,\sigma_{\text{wide}}^2$, where $m(x)$ is the slowly updated teacher's logits. At $c=1$ the prior is a tight version of DIRT-T's teacher constraint. At $c=0$ it is a zero-mean, wide prior, which implies near-uniform predictions, exactly as in NCP.
To keep dense prediction affordable, we linearize only the last layer or last block. This avoids FS-EB's full-Jacobian overhead. We also study how much accuracy and calibration that shortcut costs.
The analysis asks three questions. First, does coverage actually track the unmodeled content? We check this against labeled categories of real images, such as out-of-label-set objects and held-out lighting conditions. Second, how do the accuracy and calibration trade off as $\sigma_{\text{wide}}$ and the coverage threshold vary? Third, does broader randomization raise coverage on legitimate real images, and so recover the accuracy that gating might otherwise forfeit?
Baselines
The naive reference point is a source-only model trained on randomized synthetic data and evaluated directly on real images. For SYNTHIA→Cityscapes, DeceptionNet reports a source-only mIoU* of 22.8%, against 32.0% with its randomization. We rerun this with our backbone so that all methods share the same starting network.
The strongest competitor for accuracy on the classification benchmarks is VADA followed by DIRT-T. It is exactly the uniform entropy-minimization scheme our gating modifies, and it reports 76.5% on MNIST→SVHN and 99.6% on Synthetic Signs→GTSRB. We rerun it from the same randomized initialization with the same unlabeled target pool and the same number of optimizer steps, and give it the same hyperparameter search budget. That search covers $\lambda_e$, its adversarial-perturbation weight, and its teacher KL weight. We tune on a labeled validation split drawn only from synthetic data plus a small labeled real split that is used for model selection only, identically for every method.
For segmentation, the strongest method in the file is EHTDI. Exploring High-quality Target Domain Information for Unsupervised Domain Adaptive Semantic Segmentation reports 58.8% mIoU on GTA5→Cityscapes and 57.8% on SYNTHIA→Cityscapes (16 classes). We add our prior term to EHTDI's pipeline rather than replacing it, and compare at equal GPU hours.
No reported calibration or open-set result was found on Lacuna for any sim-to-real adaptation method in the file. For these metrics we therefore measure all baselines ourselves.
Beating the baselines means matching DIRT-T or EHTDI accuracy to within one point on covered real images. It also means clearly lowering expected calibration error on all real images and raising AUROC for rejecting out-of-label-set real content. A win on calibration alone, bought with a large accuracy loss, does not count.
Two ablations isolate the key ingredient. The first replaces simulator-derived coverage with Gaussian-noise context points, as in NCP. The second replaces it with a real-image OOD dataset, as in FSVI and FS-EB. Both keep the rest of the objective fixed. A third ablation sets $c(x)=1$ everywhere. This reduces the method to a probabilistic DIRT-T and tests whether the Bayesian treatment alone, without gating, explains any gain.
Why not just…?
Why not just threshold pseudo-labels by the model's own confidence? This is the most serious threat, because confidence-gated self-training is simple and widely used. The objection to it is that the gate is the model's own softmax. The function-space papers above show that softmax is overconfident off the training support, which is exactly where the gate needs to say no. Whether this matters in practice is empirical. We run a DIRT-T variant that gates entropy by the teacher's max-probability at a tuned threshold, with the same budget. If it matches our open-set AUROC and calibration, the coverage score adds nothing and we will report that.
Why not just ensemble or temperature-scale the adapted model? Temperature scaling needs labeled real data, which the setting forbids except for model selection. It also scales all predictions uniformly, so it cannot separate covered from uncovered inputs. Deep ensembles of DIRT-T models are a fair competitor for uncertainty. Q-SAVI's gain over deep ensembles under covariate shift (0.606 vs. 0.589 AUC-ROC) suggests function-space priors can beat them, but that evidence comes from chemistry, not vision. We train a five-member DIRT-T ensemble at equal total compute and compare directly.
Why not just randomize harder and skip target adaptation? DeceptionNet shows randomization alone reaches 90.4% on MNIST-M without real images, so harder randomization may shrink the gap enough that adaptation is unnecessary. Randomization alone gives no mechanism for flagging real content outside what it covers. We include the randomized source-only model as a baseline for both accuracy and open-set AUROC, and we sweep randomization strength to see whether its accuracy approaches that of the adapted models.
Why not use weight-space Bayesian networks or plain weight decay? The file's evidence is consistent here. FS-EB reports parameter-space MAP staying overconfident under severe corruption where function-space regularization does not. Output regularization beat weight regularization in A Speaker Adaptive DNN Training Approach for Speaker-Independent Acoustic Inversion (r = 0.700 vs. 0.682). We include a mean-field weight-space variant as a sanity check rather than a main competitor.
Proposed Experiments and Decisive Checks
The classification suite is MNIST→MNIST-M, MNIST→SVHN, and Synthetic Signs→GTSRB, plus MNIST-COCO as an unseen-background test. To make "unmodeled content" measurable, we construct open-set variants of each benchmark. In each, some real classes are absent from the synthetic label set, for example two digit classes rendered only in real SVHN or a set of GTSRB sign types with no synthetic counterpart. For segmentation we use GTA5→Cityscapes and SYNTHIA→Cityscapes. On SYNTHIA→Cityscapes, the Cityscapes classes absent from SYNTHIA's 16-class setting serve as natural uncovered content.
We report the following metrics:
- Accuracy or mIoU, overall and split by coverage decile.
- Expected calibration error.
- Selective accuracy under rejection, following FS-EB's protocol.
- AUROC for detecting out-of-label-set real inputs.
- Wall-clock cost, relative to the baselines.
Three results would settle the question. The first concerns the coverage score itself. On open-set variants, the score must separate out-of-label-set real images from in-label-set ones better than the teacher's softmax confidence does. If it does not, the method has no reason to exist.
The second is the MNIST→SVHN stress test. Here coverage may be low for most legitimate images, and gating could freeze adaptation and lose DIRT-T's accuracy. We will report that honestly. We then test whether wider randomization restores coverage and accuracy together.
The third is the segmentation check. Our term is added to EHTDI with last-layer linearization. It must keep mIoU within about a point of 58.8% on GTA5→Cityscapes. It must also lower pixel-level calibration error, especially at object boundaries, where EHTDI identifies pseudo-label corruption, and on classes absent from the source. If the gain disappears when coverage is replaced by noise or by an unrelated OOD dataset, the simulator is doing no work, and the proposal's central claim fails.
References
- A DIRT-T Approach to Unsupervised Domain Adaptation
- Noise Contrastive Priors for Functional Uncertainty
- Tractable Function-Space Variational Inference in Bayesian Neural Networks
- Function-Space Regularization in Neural Networks: A Probabilistic Perspective
- Drug Discovery under Covariate Shift with Domain-Informed Prior Distributions over Functions
- Bridging the Sim-to-Real Gap with Bayesian Inference
- DeceptionNet: Network-Driven Domain Randomization
- Exploring High-quality Target Domain Information for Unsupervised Domain Adaptive Semantic Segmentation
- A Speaker Adaptive DNN Training Approach for Speaker-Independent Acoustic Inversion