Action recognizers trained on cheap simulated video break down on real cameras. The usual fix is adversarial domain adaptation: a second network (the "domain discriminator") tries to tell simulated clips from real ones, and the main network is trained to fool it. This works poorly for human actions. In simulation versus reality, the easiest giveaway is the scene: rendered textures, lighting and backgrounds. Fooling the discriminator therefore mostly means erasing scene information. Sometimes that is exactly right, because real video models lean heavily on backgrounds as shortcuts. But scene and object context is sometimes the only way to tell actions apart, as with "drinking" versus "eating." We propose to stop guessing which features are nuisance. A simulator can render the same body motion under many different scenes, lights and camera styles. From those paired renderings we measure which feature directions change when only the scene changes. We then apply adversarial alignment only to the remaining, scene-invariant part of the representation. Scene context stays available to the classifier, and a separate penalty on motionless real clips keeps it from becoming a shortcut.
In two sentences: the simulator tells us, through controlled re-rendering, which features are "scene" and which are "motion." Sim-to-real alignment is then applied only to the motion features, so the model gains the motion-centric advantage of synthetic data without being forced to throw away real context it sometimes needs.
Core Claim
Existing synthetic-to-real action adaptation treats the gap as one undifferentiated shift. Synthetic-to-Real Domain Adaptation for Action Recognition introduces the RoCoG-v2 benchmark, which pairs Unity-simulated gesture clips with real ground and aerial video. It applies domain-adversarial training (DANN) and contrastive class alignment (CO²A) to the whole feature vector. It reports high variance for DANN and a large remaining gap on combined shifts. This paper motivates the proposal: it documents the failure but offers no diagnosis.
The closest method is TranSVAE. It splits video features into a static factor and a dynamic factor, using temporal shuffling and a mutual-information penalty, and aligns only the dynamic factor across domains. We borrow its central idea of aligning only part of the representation. We differ in how the part is chosen. TranSVAE defines nuisance as "whatever is constant over time." That definition misfires in simulation-to-reality transfer for two reasons:
- Some rendering artifacts vary over time, such as idealized trajectories and the absence of motion blur or compression.
- Some static content is genuinely discriminative, such as the cup being held.
Our nuisance subspace is instead defined by intervention: we re-render the same kinematics and record which features move. Only a simulator can supply this signal, and real-to-real adaptation methods cannot use it.
The claim is testable in a specific way. On classes where context is irrelevant, such as the RoCoG-v2 gestures, the method should match or beat full-feature alignment. On classes where context is necessary, it should clearly beat full-feature alignment. On background-swap diagnostics, it should show less scene reliance than source-only training.
Research Program
Paired renderings. For each synthetic action clip we render $m$ variants that share skeleton trajectories and camera path but differ in background, lighting, textures and simulated sensor noise. Where the original simulator cannot be re-run, we build the pairs by compositing. We take the actor, together with any hand-held object, using the simulator's exact segmentation masks. We paste it onto varied background plates, including unlabeled frames from the real target domain.
These masks are exact, which matters. HAT shows that off-the-shelf segmenters routinely assign manipulated objects to the background. Exact masks avoid that failure, and we can choose whether an object belongs to the "action" or the "scene."
Estimating the scene subspace. Let $f(x)\in\mathbb{R}^d$ be the backbone feature of clip $x$. Let $(x, x')$ be two renderings with identical kinematics and different scenes. We compute the scene-change covariance
$$C = \mathbb{E}_{(x,x')}\big[(f(x)-f(x'))(f(x)-f(x'))^\top\big],$$
and take its top-$k$ eigenvectors as the scene subspace. $P$ denotes the projector onto the orthogonal complement, which we call the motion subspace. Because $f$ changes during training, we re-estimate $C$ on a held-out batch of pairs every few hundred steps, with a moving average for stability. The choice of $k$ is set by the fraction of pair-difference variance explained, so $k$ is not an extra free parameter.
Training objective. The model is trained with
$$\mathcal{L} = \mathcal{L}_{cls}^{src}(f) - \lambda\,\mathcal{L}_{dom}(D(Pf)) + \mu\,\mathcal{L}_{static}^{tgt}(f).$$
The three terms are:
- $\mathcal{L}_{cls}^{src}$ is ordinary cross-entropy on labeled synthetic clips, computed on the full feature so that context is available to the classifier.
- $\mathcal{L}_{dom}$ is the domain discriminator loss. The discriminator $D$ sees only the motion projection $Pf$, and its gradient is reversed into the backbone with a gradient reversal layer, in the style of DANN.
- $\mathcal{L}_{static}^{tgt}$ is the ALBAR penalty applied to unlabeled real clips turned motionless by repeating one frame. It pushes predictions on those clips toward maximum entropy, with ALBAR's gradient penalty for stability. ALBAR supplies this technique and uses it within a single real dataset. We use it across domains, as the brake that stops the unaligned scene channel from becoming a shortcut on real video.
Diagnosis before method. We first test the premise that the sim-versus-real discriminator mostly keys on scene. We train a plain discriminator on source-only features. We then measure its accuracy on three input types: actor-only composites, background-only composites, and features projected onto the scene subspace versus the motion subspace. If the discriminator relies about equally on motion, the premise is wrong and the project changes. In that case, restricting alignment would buy nothing.
Baselines
Naive references. The first reference is source-only training on synthetic data with standard domain randomization. The ceiling is target-only training on labeled real data. Both use the same I3D and X3D backbones as RoCoG-v2. The file supplies the numbers we must reproduce before comparing anything:
- Ground synthetic-to-real (I3D): source-only 80.3% top-1, target-only 83.0%.
- Air synthetic-to-real (X3D): source-only 34.5%, target-only 70.3%.
All of these are reported in Synthetic-to-Real Domain Adaptation for Action Recognition.
Strongest published sim-to-real baselines. DANN and CO²A are the strongest published methods on this benchmark. The same paper reports CO²A improving air-to-air synthetic-to-real accuracy by roughly 29 points over source-only. The file gives only this approximate gain and no exact final figure. For ground-to-air, DANN is reported as a substantial but high-variance improvement, with no single number given. We rerun both with the authors' actor cropping, on the same data splits and the same number of training steps. Each method, including ours, gets the same hyperparameter search budget of twenty trials. Results are reported over five seeds with standard deviations, because variance is itself one of the failure modes at issue.
Strongest disentanglement competitor. TranSVAE reports 93.37% on UCF↔HMDB and 52.6% on Epic-Kitchens. No reported result was found on Lacuna for TranSVAE on RoCoG-v2 or any photorealistic synthetic-to-real action benchmark, so we run it ourselves under the same budget. Beating it means higher mean target accuracy with overlapping-or-lower variance on RoCoG-v2. It also means a clear margin on the context-dependent split described below.
Ablations isolating the key ingredient. We run four variants:
- Full-feature DANN plus ALBAR's static penalty. This tests whether the subspace restriction matters.
- Our method with a random $k$-dimensional subspace in place of the simulator-estimated one. This tests whether the paired renderings carry real information.
- Our method with the scene subspace estimated from TranSVAE-style temporal shuffling instead of re-rendering. This isolates intervention versus temporal invariance.
- Our method without $\mathcal{L}_{static}^{tgt}$. This checks whether the preserved scene channel turns into a shortcut.
Why not just…?
Why not just mask out the background in both domains? The simulator gives perfect masks, so this is cheap on the synthetic side. The file's evidence nonetheless says it caps performance on context-dependent actions:
- Seeing Beyond the Scene finds segmentation reduces background bias by 3.78% but hurts when context is needed.
- In BEAR, fully context-free skeleton models drop to 30.24% versus 59.90% for RGB on environment-aware behaviors.
- RoCoG-v2 already crops the actor on real video and still shows a roughly 36-point source-to-target gap on ground-to-air.
On the gesture task, masking may be competitive. We include masked source-only and masked DANN as baselines, and expect them to lose specifically on the context-dependent split.
Why not just run TranSVAE, which already aligns only the dynamic factor? This is the most serious threat. If temporal invariance already separates rendering style from kinematics, re-rendering adds cost for no gain. The shuffling-based subspace ablation decides this directly. If it matches the simulator-estimated subspace on both RoCoG-v2 and the context split, the intervention idea is unnecessary. The honest conclusion would then be "TranSVAE plus ALBAR suffices."
Why not just add a paired-rendering consistency loss plus ordinary DANN? Making synthetic features identical across scenes is simpler and uses the same pairs. The problem comes after that: full-feature DANN then drives real features to discard scene information too, because synthetic features no longer carry any. That is the context-stripping failure again, arrived at by a different route. We run this variant under the same budget. If it matches us on context-dependent classes, our claim about preserving context is false.
Why not just pretrain on more real video? RESOUND finds Kinetics among the most static-biased datasets. HAT finds Kinetics models retain about 70% of accuracy with the human removed. SynAPT shows synthetic pretraining edging out Kinetics on Diving48 (63.50% vs 62.84%). Scale alone does not remove scene shortcuts. We still report a Kinetics-initialized source-only model as a reference.
Proposed Experiments
RoCoG-v2. We test all four synthetic-to-real directions, with the ground-to-air combined shift as the headline. Here context is nearly irrelevant, so this benchmark tests whether preserving context costs anything and whether restricting alignment reduces DANN's variance.
Context-dependent split. Second, we build a split where context matters. We select action classes shared between SynAPT's synthetic sources (ElderSim, PHAV, SURREACT) and real datasets, favoring object- or scene-dependent pairs such as eating versus drinking. We train on synthetic data and adapt to unlabeled real clips. The size of this split depends on label overlap, which we will report rather than assume.
Bias diagnostics. Third, we measure bias on the real target with HAT's background-only ratio and action-swap rates. For context: across 74 Kinetics models, HAT reports that models predicted the background's class 29.5% of the time versus 16.8% for the human's action. We also run the shuffled-frame test from Rethinking Training Data, and evaluate on Mimetics, which HAT reports correlates at 0.8577 with its synthetic evaluation.
Decisive checks. The project succeeds if four things hold together:
- The premise diagnostic shows the discriminator relies mainly on the scene subspace.
- The simulator-estimated subspace beats both the random and the shuffling-based subspaces.
- Our method beats full-feature DANN plus consistency on the context-dependent split, while matching or beating CO²A and TranSVAE on RoCoG-v2 with lower seed variance.
- Background-only accuracy on real targets falls relative to source-only, which shows that the preserved scene channel has not become a shortcut.
A failure on check 2 or 3 would mean the simulator's counterfactual power does not translate into better alignment. We would report that as the finding.
References
- Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performances
- Unsupervised Video Domain Adaptation for Action Recognition: A Disentanglement Perspective
- Enabling Detailed Action Recognition Evaluation Through Video Dataset Augmentation
- ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition
- Seeing Beyond the Scene: Analyzing and Mitigating Background Bias in Action Recognition
- BEAR: A Video Dataset For Fine-grained Behaviors Recognition Oriented with Action and Environment Factors
- RESOUND: Towards Action Recognition Without Representation Bias
- How Transferable are Video Representations Based on Synthetic Data?
- Rethinking Training Data for Mitigating Representation Biases in Action Recognition