A simulator gives you two kinds of information about every training example. The first is what caused the outcome: the commanded action, the initial object pose, the physical setup. The second is what the outcome caused: the camera image or force reading recorded afterward. When a model trained in simulation fails on a real robot, the usual response is to treat these inputs as one block and align simulated and real feature distributions. This proposal uses the split instead. Often the cause-side data carry over from simulation almost unchanged, because a commanded push is the same command in both worlds. What breaks is the sensor side, since rendered images do not look like camera images. If that is the case, a predictor of the outcome from causes alone, learned in simulation, can serve as a per-example prior on the label of each unlabeled real example. Real effect data can then be fit as a mixture whose mixing weights are known and vary across examples. That turns unlabeled real rollouts into a well-posed estimation problem rather than a guess-and-confirm pseudo-labeling loop.
The second contribution is a diagnostic. Holding the simulated cause-to-outcome model fixed is a testable assumption, so the same machinery can tell whether a given sim-to-real gap lives in the physics, in the sensors, or in both. This speaks directly to an open contradiction. Causal Feature Alignment shows that discarding spurious image features helps out-of-distribution accuracy. In contrast, Nastl and Hardt find that models using all features beat causal-only predictors on 16 of 16 real tabular shift tasks. Our method discards nothing. Cause features act as a prior and effect features act as evidence. The diagnostic predicts when each kind of feature should carry the prediction.
Core Claim
Write $c$ for the cause features (action, initial state), $y$ for a discrete outcome (for example push succeeded, object rotated, or object fell), and $x$ for the effect features (post-action image, force-torque trace). In simulation we learn a cause model $\pi_y(c)=P_{sim}(y\mid c)$. We assume that the real domain changes only the effect mechanism $q_y(x)=P(x\mid y)$. Under that assumption, unlabeled real pairs satisfy
$$P_{real}(x \mid c) = \sum_{y} \pi_y(c)\, q^{real}_y(x),$$
so each real example is a draw from a mixture with known, example-specific weights. If the matrix of weight vectors $\{\pi(c)\}$ over the observed causes has rank equal to the number of outcome classes, the components $q^{real}_y$ are identified. Each region of cause space gives one linear equation in the unknown components, which is the same argument that makes learning from label proportions work. Ordinary self-training has no such anchor. Its only prior is the classifier's own confidence, which is the source of the confirmation bias seen in Syn2Real, where adapted detectors stayed at 0.02–0.04 mAP.
The idea that differs from prior work is the assumed shift. Semi-Generative Modelling uses the same factorization, but it assumes covariate shift, meaning both mechanisms are shared and only $P(c)$ moves. It was also tested only on toy, LUCAS and Sachs data. We assume the opposite kind of shift: a single mechanism moves, and it is the effect mechanism, which is the natural model of a rendering or sensor gap. The conditional cluster assumption supplies the intuition that labels are constant within clusters of $P(x\mid c)$. Its conditional self-learning (52.6% to 71.9% on UCI Heart Disease) assumes additive-noise maps learned on one domain and has never met a domain gap.
Research Program
The method has three parts.
Cause model. We train $\pi$ on simulated rollouts. Because it sees only actions and states, it has no access to rendering artifacts.
Real effect model. On the real side we train a discriminative effect classifier $r_\theta(y\mid x)$ with an encoder initialized from simulation. Training uses an expectation-maximization loop. In the E-step, the posterior for real example $i$ is proportional to $\pi_y(c_i)\,r_\theta(y\mid x_i)/P_{sim}(y)$. In the M-step, $r_\theta$ is fit to these soft targets, which is equivalent to maximizing the mixture likelihood above. The prior stays fixed throughout, so the loop cannot drift toward a degenerate labeling the way unanchored self-training can. If the effect mechanism depends on the causes (for example, appearance depends on object pose), $c$ is also fed to $r_\theta$, and identifiability then needs variation of $\pi(c)$ within local neighborhoods. We will measure this rather than assume it.
Diagnostic and second stage. If physics also shifted, the frozen-prior model misfits in a detectable way. Within regions of cause space, the average real posterior departs from $\pi(c)$, and held-out marginal likelihood drops relative to a model with a free prior. We calibrate a mechanism-shift test on these signals. When the test fires, we run a second stage. The adapted effect classifier labels the real rollouts, and those inferred labels are used to refit the cause model on real data. That refit corrects the physics side for planning. When both mechanisms shift at once, the problem is not identifiable from unlabeled data. We say this openly and measure how many real labels are needed to anchor it.
Testbeds
Controlled factorial testbed. We start from a manipulation simulator. The "real" domain is a second simulator configuration with a separately chosen renderer and sensor-noise model, perturbed physics parameters, or both. Ground truth about which mechanism moved is known, so both identifiability and the diagnostic can be checked exactly.
Physical tasks. The physical tasks follow the Kinova Gen 3 setups from COMPASS (Push-I with 67 parameters and the air-hockey task with 64). Autonomous rollouts provide unlabeled $(c,x)$ pairs. A held-out set is outcome-labeled with motion capture or human annotation.
Image-only benchmarks. We deliberately do not apply the method to VisDA-C. Pure image classification has no observed cause channel, so the method reduces to ordinary clustering there. Including it would only obscure the claim.
Baselines
Naive reference: source-only training. The naive reference point is a predictor trained in simulation on all features, $[c,x]$, and deployed unchanged. This is the setting where VisDA reports AlexNet falling from 99% to 28%. Beating it is necessary but uninformative.
Kitchen-sink model with distribution alignment. This is one network on all features, aligned across domains with a Deep Adaptation Network loss (MMD), CORAL, or an adversarial domain discriminator. It is the direct embodiment of the Nastl–Hardt position that using every feature is enough. On VisDA-C, DAN reached 51.62%. For our tasks, no reported result was found on Lacuna.
Mean-teacher self-training on all features. This is the method family that won VisDA-C at 92.8% and reached 91.8% on Syn2Real closed-set. It is the strongest generic unsupervised adaptation competitor, and it uses exactly the same unlabeled real rollouts we do. Our method beats it only if we show higher held-out real outcome accuracy and better calibration at equal rollout count. For our tasks, no reported result was found on Lacuna.
Covariate-shift semi-generative model. This is Semi-Generative Modelling with both mechanisms refit jointly, plus conditional self-learning. These are the closest causal competitors. Comparing against them isolates our assumption about which mechanism is allowed to shift.
COMPASS for the downstream policy. For downstream control, the strongest method is COMPASS. It discovers which simulator parameters cause trajectory discrepancy and tunes them, and it reports an 80% real air-hockey success rate against 39% for the best baseline and 0% for the nominal simulator. We will rerun its released pipeline with the same number of real rollouts and the same tuning budget. COMPASS needs trajectory tracking; our method needs only cameras and logged actions. Matching its success rate under that constraint would count as a win, and exceeding it when the gap is mainly visual would count as a clear win.
Ablations. Three ablations isolate the key ingredient. First, a uniform prior replaces $\pi(c)$, which reduces the method to self-training on effects. Second, the frozen side is swapped: we fix the effect model and adapt the cause model. Third, the prior is shuffled across examples, which tests whether gains come from per-example weights or merely from class proportions. Every method gets the same encoder, the same hyperparameter search budget, and model selection on the same small labeled real validation set.
Why not just…?
Why not just run mean-teacher self-training on all features? This is the real threat, because it often works on classification. Its weakness is that the pseudo-label prior is the model's own output, so errors made under a rendering gap reinforce themselves. That is the failure Syn2Real documents for detection and open-set classification. The uniform-prior ablation is exactly this comparison with everything else held fixed. If it matches the full method in the render-only factorial cells, our core claim fails and we will report that.
Why not just use every feature with domain alignment, as Nastl and Hardt's results suggest? Their finding concerns discarding non-causal features, and we keep all of them. The open question is whether marginal alignment can find the right correspondence when the effect mechanism itself changes. Alignment matches $P(x)$ without regard to $y$, whereas our objective matches $P(x\mid c)$ with known mixing weights. The factorial testbed shows directly whether alignment suffices when only the sensor moved.
Why not just tune the simulator (system identification, COMPASS, or domain randomization) and retrain? Simulator tuning attacks the physics side well, as COMPASS's 80% success shows. Rendering and sensor gaps, however, are not low-dimensional parameter problems. Syn2Real also found that adding unstructured visual clutter hurts transfer. The diagnostic gives a concrete answer here. In cells where only physics shifted, we expect COMPASS to win and our test to say so. The two methods should be complementary, and we will test that combination.
Why not just label a few real rollouts? Few-label fine-tuning is the honest upper reference. We will report full label-efficiency curves. The method is worthwhile only if it reaches the same accuracy with substantially fewer real labels.
Decisive Checks
Check 1: render-only shift. In the factorial testbed, the full method should recover the effect components and beat mean-teacher self-training and alignment only in the render-only cells. Its advantage should shrink as the cause-prior matrix loses rank, which we induce by restricting action diversity. If the advantage does not track rank, the identifiability story is wrong.
Check 2: the diagnostic. The mechanism-shift test should separate render-only, physics-only and joint shifts. We will report its ROC curve against ground truth. On the Kinova tasks, the test's verdict should predict which baseline wins. When it reports a sensor gap, our adapted effect model should beat COMPASS at equal rollout budget. When it reports a physics gap, COMPASS should win.
Check 3: when causes alone suffice. As a direct test of the Nastl–Hardt contrast, the cause-only predictor should match full-feature models exactly in the cells where the test reports a sensor-only gap, and lose elsewhere. The main risk, besides joint shift, is an unobserved confounder in the real world, such as lighting that correlates with outcome. That would show up as misfit the diagnostic cannot attribute to either mechanism, and we will report those cases separately.
References
- Causal Feature Alignment: Learning To Ignore Spurious Background Features
- Do causal predictors generalize better to new domains?
- Syn2Real: A New Benchmark forSynthetic-to-Real Visual Domain Adaptation
- Semi-Generative Modelling: Covariate-Shift Adaptation with Cause and Effect Features
- Semi-supervised learning, causality, and the conditional cluster assumption
- What Went Wrong? Closing the Sim-to-Real Gap via Differentiable Causal Discovery
- VisDA: The Visual Domain Adaptation Challenge