A model trained on rendered images and deployed on real photographs often loses half or more of its accuracy. Papers that close part of this gap usually name one cause and fix it: the texture is too clean, the image frequency statistics are wrong, the hands have no forearms. Nobody reports what fraction of the drop each named cause is responsible for, or how much is left over after all of them. Epidemiology has faced this problem for decades. A "mediator", meaning an intermediate variable through which a cause acts, can be statistically significant and still carry only 5–30% of the effect. Mendelian randomization studies, for example, find that socioeconomic status carries about 10% of the link between schizophrenia and depression (Identifying the mediating role of socioeconomic status...), and FGF5 carries under 10% of lipid-lowering drugs' effect on atrial fibrillation (Lipid-lowering drugs, circulating inflammatory factors, and atrial fibrillation). Sim-to-real research has never produced the matching number.
We propose to compute it. The key difference from epidemiology is that in sim-to-real work we can intervene on the mediators directly instead of inferring them from observational data. We can swap a real image's Fourier amplitude spectrum for a synthetic one, restyle its texture, or add or remove occluders. Each swap measures how much of a frozen synthetic-trained model's error on real data flows through that factor, without any retraining. What remains after all named factors are swapped is the unexplained mediation variance of the sim-to-real gap. The central claim to test is that this accounting predicts in advance how much each factor-specific fix (frequency augmentation, pixel translation, test-time adaptation) can recover, and that a large residual reliably signals a missing cause.
Core Claim
Let $D \in \{\text{sim}, \text{real}\}$ be the image domain, and let $L$ be the task loss of a fixed model trained only on synthetic data. The gap is $G = \mathbb{E}_{\text{real}}[L(x)] - \mathbb{E}_{\text{sim}}[L(x)]$. For a named factor $k$, let $T_k^{\text{sim}}(x)$ be an operator that replaces factor $k$ of a real image $x$ with a value drawn from the synthetic distribution and leaves everything else in place. Examples are the amplitude spectrum with the phase kept, texture statistics via a learned translator, or an occlusion layout. The share of the gap mediated by $k$ is then
$$ s_k = \frac{\mathbb{E}_{\text{real}}[L(x)] - \mathbb{E}_{\text{real}}[L(T_k^{\text{sim}}(x))]}{G}. $$
This is the analogue of a natural indirect effect, where the treatment is the domain and the mediator is the factor. The reverse operator $T_k^{\text{real}}$ applied to synthetic images gives a second estimate from the other side. Applying all operators jointly gives $s_S$ for the full factor set $S$. The residual $r = 1 - s_S$ is the part of the gap no named factor accounts for. Factors interact, so we report a four-way-style split into pure mediated shares and interaction terms, borrowing the decomposition used in Vascular-related biological stress, DNA methylation, allostatic load. We also report Shapley averages over orderings.
The claim makes two falsifiable predictions. First, a fix that targets factor $k$ should recover roughly $s_k G$ of the gap and not much more. Second, benchmarks with large $r$ should be the ones where every factor-level fix stalls.
Relation to Prior Work
Four pieces of prior work motivate this proposal because they found the gap without measuring its composition. VisDA documents a source-only AlexNet falling from 99% on CAD renders to 28% on real images, but treats that drop as a single number. Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation identifies amplitude mismatch and missing forearms and occlusion as causes, and fixes them to reach 1.02 cm joint error against 0.99 cm for real-data training. Its decomposition is heuristic: it shows the fixes work but never apportions the original gap among them. PASTA shows that high-frequency amplitude perturbation raises GTAV→Cityscapes segmentation from 28.95 to 44.12 mIoU, again without saying what fraction of the gap amplitude carries. Transferring disentangled representations argues that shared "structural similarity" between domains matters more than style. That is a hypothesis about mediation which our accounting can test directly: structure-level factors should carry more of $s_S$ than style-level ones.
Two lines of work supply the method. The activation-ablation accounting in Decoded but Unused, where ten attention heads carry 59% of a routing effect and 41% is left over, is the reporting standard we adopt: always state the explained fraction and the residual. Beyond Surface Structure separates surface from deep causal paths in language models, which is the same split we make between appearance factors and geometric or structural ones. We differ from both by working on visual domain shift rather than text framing, and by intervening on the input image rather than only on internal weights.
The remaining related work constrains the design or solves a different problem. DMAVAE deconfounds mediation analysis with a partitioned latent space and cuts direct-effect bias from over 7% to 1.5% on synthetic structural-equation data. We do not need its latent deconfounding, because domain and factors are set by intervention rather than observed. Its concern still applies to us in a different form: our operators may create artifacts that act as hidden confounders. COMPASS discovers which simulator parameters cause trajectory mismatch and tunes them, raising real-world success from 39% to 80%. It identifies causes but does not apportion the gap. CoC-VLA chains perception, prediction and planning across CARLA and nuScenes but never measures how much downstream error each stage carries. That setting could be a later target for our method; we do not study it here.
Research Program
We will build a library of factor operators. Each operator is validated on two points before its numbers are trusted. The first is idempotence on its home domain: applying $T_k^{\text{sim}}$ to synthetic images must leave the loss essentially unchanged, which bounds how much the operator's own artifacts contribute. The second is two-sided agreement: the sim→real and real→sim estimates of $s_k$ should agree within a tolerance fixed in advance. An operator that fails either check is reported as unreliable rather than silently kept.
The initial library contains four operators. The amplitude operator swaps Fourier amplitudes and keeps the phase. The texture/style operator uses a PixelDA-style translator (Unsupervised Pixel-Level Domain Adaptation). The sensor-noise operator uses fitted noise models. The occlusion/geometry operator uses composited occluders, forearm addition for hands, and CAD re-rendering where real poses are known.
We add one internal mediator: batch-normalization statistics, swapped between domains. Test-time adaptation methods such as Tent act on exactly this quantity, so its share $s_{\text{BN}}$ is a direct prediction of what they can recover.
Baselines
The naive reference point is the undecomposed gap $G$ alone, the way benchmarks report it today. Against this, the proposal must show that it adds predictive information about which fix will help.
The strongest competitor is attribution by retraining. For each benchmark, train the synthetic model once per factor-specific fix and read off each fix's realized gain as the factor's importance. The fixes are PASTA-style amplitude augmentation, PixelDA translation, occlusion augmentation, and Tent at test time. This is what practitioners already do implicitly, and it measures fix efficacy directly. We run it with the same backbones (DeepLabv3+ for segmentation, as PASTA uses), the same synthetic training data, and the published hyperparameters. The retraining side gets a matched tuning budget per fix. Our accounting uses zero retraining.
The reference numbers are taken from the papers. On GTAV→Cityscapes, PASTA reaches 44.12 mIoU from a 28.95 baseline (PASTA) and Tent reaches 35.8 from 28.8 (Tent); the VisDA-S challenge winner reaches 47.5 (VisDA). On LineMod, PixelDA reaches 23.5° against DANN's 56.58°. On Sim10K→Cityscapes, PASTA reports a +16.9 mAP gain, but no reported absolute baseline found on Lacuna.
Beating this baseline means the following. On benchmarks held out from any tuning of the operators, our shares $s_k G$, computed without retraining, must rank fixes as well as retraining does. Specifically, the rank correlation with realized gains across (method × benchmark) cells must not be significantly worse. Our method must also predict gains on new benchmarks before the fixes are run there.
A second baseline is distance-based attribution. Here each factor's importance is the discrepancy (MMD, as in DAN) between sim and real distributions of that factor or of layer features. Distances ignore whether the model uses the factor, so we expect them to rank worse. That expectation is testable.
Two ablations isolate the key ingredient. The first drops the two-sided validation and uses single-direction swaps; it shows whether operator artifacts distort the shares. The second drops the joint intervention and sums the single-factor shares; it shows whether interactions, and therefore the honest residual, matter.
Why not just…?
Why not just retrain with each fix and see what helps? This is the real threat, and the retraining baseline above is designed to settle it. Retraining measures how good a particular fix is, which mixes together how much the factor matters and how well that fix handles it. It also costs a full training run per factor, and it says nothing about the part of the gap that no available fix targets. If retraining-based attribution predicts gains on held-out benchmarks as well as our accounting does, our contribution shrinks to a cheaper approximation. We will say so. The residual $r$ would still be new information.
Why not just apply PASTA or Tent and skip the diagnosis? They work but plateau at different levels: 44.12 versus 35.8 mIoU on the same GTAV→Cityscapes shift. Tent is known to reinforce overconfident errors under structural shift. The accounting predicts where each plateau sits. If $s_{\text{BN}}$ is small relative to $s_{\text{amplitude}}$ on GTAV→Cityscapes, the Tent-versus-PASTA ordering is explained rather than merely observed. If a benchmark shows large $r$, neither method is the right investment.
Why not just learn disentangled latent factors and measure their shift? Transferring disentangled representations shows that such latents transfer only under structural similarity and has so far been demonstrated on object-centric datasets. Measuring latent shift is also a distance, not a causal share. Our distance-based baseline tests this family directly.
Proposed Experiments
The first experiment is a planted-factor test with known answers. Starting from Shapes3D and Isaac3D, we construct shifts by composing known amounts of amplitude change, texture change and occlusion, so the true mediated shares are fixed by design. We also leave one factor deliberately outside the operator library, so the method must report a nonzero residual of the right size. Failing this test ends the project.
The second experiment is a published-answer test on hand pose (SynFrei→real). The cited work closed the gap to 97% of real-data accuracy by fixing amplitude and forearm/occlusion. Our accounting on the untuned synthetic model should therefore assign most of $G$ to those two factors and leave a small $r$. A large residual here would mean the operators are missing what the fixes actually addressed.
The third experiment uses paired data from LineMod, where real poses allow exact CAD re-rendering. We will produce pixel-matched sim/real twins, which make the two-sided estimates sharpest. We will check whether PixelDA's gain matches the texture share.
The fourth experiment is the prediction test on GTAV→Cityscapes, Sim10K→Cityscapes and VisDA-C. We compute shares with no retraining, register predictions for PASTA, PixelDA and Tent gains, then run the fixes. The decisive number is the agreement between predicted $s_k G$ and realized gain across all cells, compared with the retraining and distance baselines. The second outcome is whether benchmarks with larger $r$ show smaller best-fix recovery.
The metrics are mIoU, mAP, accuracy, joint error and angle error, each converted to fraction-of-gap. We add bootstrap intervals on every share, which is also how the epidemiology papers report mediated fractions.
References
- Identifying the mediating role of socioeconomic status on the relationship between schizophrenia and major depressive disorder: a Mendelian randomisation analysis
- Lipid-lowering drugs, circulating inflammatory factors, and atrial fibrillation: a mediation Mendelian randomization study
- VisDA: The Visual Domain Adaptation Challenge
- Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation
- Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization
- Transferring disentangled representations: bridging the gap between synthetic and real images
- Decoded but Unused: Instruction Tuning Routes Moral Framing into the Judgment Readout
- Beyond Surface Structure: A Causal Assessment of LLMs' Comprehension ability
- Disentangled Representation for Causal Mediation Analysis
- What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
- CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Tent: Fully Test-Time Adaptation by Entropy Minimization