Models trained on synthetic images often fail on real photographs, and nobody knows which part of the simulation they latched onto. A renderer is not a black box, though. Every synthetic image comes with a logged list of the continuous settings that produced it: light direction and intensity, material roughness, camera blur and noise, background, viewpoint. This proposal treats those settings as measurable nuisance variables. We use neural mutual information estimators, which are small auxiliary networks trained to tell true (feature, setting) pairs from shuffled ones, to measure how much each layer's features still encode each setting once the class label is accounted for. We then test two claims. The first is that this measurement, taken on synthetic data alone, predicts how badly a model will fail on real data. The second is that penalizing it during training improves real-domain accuracy more than the simpler options do.
In two sentences: a network that encodes the light angle of a CAD rendering more than it needs to will probably also encode the differences between a rendering and a photo. Because the simulator records its own settings, we can estimate and suppress that dependence directly, per setting, without any real labels and without guessing what the confounder is.
Core Claim
Let $Z$ be a feature vector, $Y$ the class label, and $\theta \in \mathbb{R}^P$ the logged nuisance settings, which are sampled independently of $Y$ by design. The quantity we estimate and penalize is the conditional mutual information $I(Z;\theta \mid Y)$. We also estimate it per setting, $I(Z;\theta_k \mid Y)$, where $\theta_k$ is the $k$-th setting.
The simulator supplies something that most shortcut-removal settings lack. Because $\theta$ is drawn from a known prior $p(\theta)$, samples from the "independent" distribution $p(z \mid y)\,p(\theta)$ can be drawn exactly: keep $(z, y)$ and draw a fresh $\theta'$ from the prior. Most estimators instead approximate this distribution by shuffling within a minibatch, which introduces bias. Fresh draws avoid that bias. They also avoid the per-class batch partitioning that makes conditional estimation sample-hungry. We use the Donsker-Varadhan bound from Mutual Information Neural Estimation with a critic $T_\omega(z,\theta,y)$:
$$\hat I = \mathbb{E}_{(z,\theta,y)}[T_\omega(z,\theta,y)] - \log \mathbb{E}_{(z,y),\,\theta' \sim p(\theta)}[e^{T_\omega(z,\theta',y)}]$$
The training objective is source cross-entropy plus $\lambda \hat I$, with the critic maximizing $\hat I$ and the encoder minimizing it. The dependence we want to remove should be small to moderate, which is the regime where this bound's variance is manageable. The known failure of the bound at large mutual information (MINE) therefore matters less here than in representation-learning uses that maximize mutual information.
What would be new is the conditioning variable: continuous, multi-dimensional, and exactly known simulator settings, in place of a binary domain flag or an annotated confounder. Gap 1 of the research file identifies this as open. The per-setting profile also gives a label-free diagnostic, which matters because sim-to-real practitioners currently have no principled way to choose checkpoints or hyperparameters without real labels.
Relation to Prior Work
Three papers motivate the approach by showing that neural mutual information penalties beat adversarial alternatives when the confounder is labeled.
Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging compares MINE, distance correlation and gradient-reversal classifiers on CheXpert with an inverted sex confounder. On that inverted test set, the baseline falls to 45.8 AUROC and MINE reaches 87.7.
Information-Theoretic Bias Reduction via Causal View of Spurious Correlation penalizes $I(F;Z \mid Y)$ with MINE. It raises Waterbirds worst-group accuracy from 52.3% to 72.2%.
Nuisance-extended Information Bottleneck splits features into task and nuisance codes and enforces their independence adversarially.
All three need a discrete, annotated confounder. We replace it with the simulator's own continuous settings. The Cobias paper's conditional estimator also requires partitioning batches by class; our exact prior sampling removes that requirement.
Two lines of work supply contrast on the sim-to-real side.
Domain Adaptation via Maximizing Surrogate Mutual Information (SIDA) maximizes class-conditional mutual information between source and target features through a pseudo-labeled surrogate distribution and reaches 84.0% on VisDA-2017. It uses a binary domain notion and depends on target-manifold quality. We never touch target labels or pseudo-labels for the penalty.
Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective and Leveraging Mutual Information for Asymmetric Learning under Partial Observability do use simulator parameters. However, they treat them as privileged, task-relevant state to be recovered: the first with a cosine-similarity surrogate, the second with mutual information over low-dimensional states. We treat a disjoint set of settings as nuisance to be suppressed, in high-dimensional visual features.
MIC is the strongest visual adaptation method in the file. Its masked-consistency loss uses unlabeled real images and is complementary to our approach, so we test the combination.
One finding constrains the design. Adversarially Masking Synthetic to Mimic Real reports that domain discriminators latch onto trivial clean-versus-noisy cues. This warns that any learned critic, ours included, can lock onto a single easy setting, and it is one reason we report per-setting estimates rather than a single number.
Research Program
We build a renderer over the 12 VisDA-C object classes from CAD models. For every image it logs lighting azimuth, elevation and intensity, material roughness and albedo jitter, a background index (embedded as a vector), defocus blur, sensor noise level, and camera viewpoint. All settings are sampled independently of class. The real target is the VisDA-2017 COCO validation split described in VisDA.
The work proceeds in three stages.
- Calibrate the estimator. We inject $\theta$ into frozen features through a Gaussian channel whose mutual information is known analytically. We then check that per-setting estimates track the true value and rank settings correctly at the batch sizes we will train with.
- Run the diagnostic study. We train a pool of roughly 60 models that vary backbone, augmentation, randomization width, regularization and training length. For each model and layer we compute the per-setting profile on held-out synthetic data and correlate it with real-target accuracy.
- Run the intervention. We train with the penalty applied to the full $\theta$ and, separately, only to the settings the diagnostic flagged. We evaluate alone and on top of MIC.
As a secondary, lower-cost check of generality, we repeat the diagnostic on DeepMind Control tasks with randomized observation-noise and latency settings, keeping the friction and mass settings that HIB treats as privileged out of the penalty. No reported numeric result was found on Lacuna for this benchmark, so it serves only as a within-study comparison.
Baselines
The naive reference point is source-only training with the same domain randomization over $\theta$, on the same backbone and with the same synthetic data. It shows whether randomization alone already makes features independent of the settings. If our estimates are near zero for this model, the premise is wrong.
The strongest existing method is MIC, run from the official code on our synthetic set with the same backbone and schedule. Its paper reports 92.8% VisDA-2017 classification accuracy (MIC). That figure was obtained on the official VisDA synthetic data, so we re-run MIC rather than compare against the quoted number. We also report SIDA's published 84.0% (SIDA) as context. Our penalty uses no target images, so beating MIC alone is not the bar. The bar is that MIC plus the penalty exceeds MIC by more than seed variance across five seeds, and that the penalty alone closes a meaningful share of the gap between source-only training and MIC.
Two competitors use the logged settings without mutual information. The first is a gradient-reversal regressor that predicts $\theta$ from $Z$ and pushes features to defeat it. The second is an HSIC or distance-correlation penalty between $Z$ and $\theta$.
The third competitor is paired re-render consistency: render the same scene under two values of $\theta$ and pull the features together.
All these competitors get the same synthetic images, the same total compute including critic updates, and the same 20-run hyperparameter search over penalty weight. Two ablations isolate the key ingredient. The first replaces exact prior resampling with in-batch shuffling. The second drops the conditioning on $Y$.
Why not just…?
Why not just randomize the simulator more widely? Wider randomization is exactly what the do-nothing baseline tests. Driving in the Matrix shows that scaling synthetic data helps but leaves a residual gap. Randomization makes $\theta$ independent of the label, but nothing stops the encoder from representing $\theta$ anyway. The diagnostic study measures directly whether wider randomization drives $I(Z;\theta_k \mid Y)$ to zero. If it does, the penalty is redundant and we will say so.
Why not just consistency across paired re-renders? This is the most serious threat. Pairwise consistency approximately minimizes how much features move along $\theta$ for a fixed scene, which overlaps heavily with what we penalize. Our argument for mutual information rests on two points. First, it needs no paired renders, so it applies to logged synthetic datasets that cannot be re-rendered. Second, it measures dependence over the whole distribution rather than along sampled pairs, and the shuffled-pair term in the estimator prevents the collapse that positive-only similarity invites (Deep InfoMax). If paired consistency matches us in accuracy and as a diagnostic when pairs exist, we narrow the claim to the unpaired setting.
Why not just an adversarial regressor or HSIC on the settings? Both are cheaper. The CheXpert benchmark (Benchmarking Dependence Measures) found MINE beat both adversarial classifiers and distance correlation on high-dimensional features, but only for a binary confounder. Whether that ranking holds for continuous multi-dimensional settings is precisely what our same-budget comparison settles.
Decisive Checks
The diagnostic claim succeeds if the Spearman correlation between estimated $I(Z;\theta \mid Y)$ and real-target error across the model pool is clearly higher than for the three label-free alternatives: source validation accuracy, domain-discriminator accuracy, and paired-render feature sensitivity. A second success criterion is that selecting the lowest-mutual-information checkpoint recovers most of the oracle checkpoint's real accuracy.
The intervention claim succeeds if, at matched compute, the penalty beats source-only, the gradient-reversal regressor, HSIC and the unpaired-equivalent consistency variants on VisDA real accuracy across five seeds, and adds to MIC rather than overlapping with it. Penalizing only the diagnostically flagged settings should match or beat penalizing all of them, which shows the profile carries real information.
The claim fails if randomization alone already yields near-zero estimates, if paired-render sensitivity predicts transfer as well as the mutual information estimate does, or if the in-batch-shuffling ablation performs identically. That last outcome would mean exact prior sampling is not what makes the method work.
References
- Mutual Information Neural Estimation
- Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging
- Information-Theoretic Bias Reduction via Causal View of Spurious Correlation
- Enhancing Multiple Reliability Measures via Nuisance-Extended Information Bottleneck
- Domain Adaptation via Maximizing Surrogate Mutual Information
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- Leveraging Mutual Information for Asymmetric Learning under Partial Observability
- MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- VisDA: The Visual Domain Adaptation Challenge
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Learning deep representations by mutual information estimation and maximization