Video simulators can produce unlimited labeled clips of people performing gestures and actions, but recognizers trained on those clips often fail on real cameras. One reason is that a video model can classify a clip from what the scene looks like (terrain texture, lighting, the rendered character) rather than from how the body moves. Recent video domain-adaptation methods try to fix this by splitting a clip's features into a "static" part, meaning appearance that does not change over time, and a "dynamic" part, meaning the motion. They then align only the dynamic part between training and deployment data. These methods have been built and tested only on transfer between two real datasets, where nobody knows which features are appearance and which are motion. A simulator removes that uncertainty. It can render the exact same motion-capture sequence under many different appearances. That gives direct, free supervision for the split, and real-to-real methods have never been able to use it.
We propose to use such render pairs to supervise the content-motion split, and to apply an adversarial alignment step only to the motion features that remain. The bet is that in synthetic-to-real video the appearance gap can largely be removed by supervised disentanglement learned inside the simulator. What then remains is a smaller gap in kinematics: robotic timing, overly smooth trajectories, no camera shake. A class-aware adversary can handle that remaining gap without erasing the motion cues that separate actions. The test bed is RoCoG-v2, a benchmark where standard adversarial adaptation is known to be unstable and to confuse temporally opposite gestures.
Core Claim
When a synthetic training set contains multiple renders of the same underlying motion, a constraint that forces the motion features of those renders to be identical produces a cleaner content-motion split than either unsupervised route. The first unsupervised route is the loss-heavy generative approach of TranSVAE. It decomposes video with a sequential variational autoencoder, a model that learns a compressed code by reconstructing its input. It is regularized by mutual-information, contrastive and adversarial terms. The second route is the architectural approach of MetaTrans. It subtracts the output of an order-blind encoder from an order-aware encoder and uses only two losses. The cleaner split should matter most on compound shifts, where synthetic-to-real and ground-to-aerial viewpoint changes occur together. It should show up in three ways: higher target accuracy, lower variance across random seeds, and fewer confusions between gestures that differ mainly in temporal direction.
The proposal differs from the closest work as follows. MetaTrans and TranSVAE have to infer the static/dynamic split without supervision, because real-to-real benchmarks give no paired evidence. We supervise it directly, which only simulation makes possible. In the other direction, RoCoG-v2 works in the synthetic-to-real setting but uses monolithic I3D/X3D backbones that do no disentangling.
Research Program
The prior work plays three roles here.
Some of it motivates the problem. Rethinking Training Data for Mitigating Representation Biases in Action Recognition shows that 3D CNNs trained on generic data act almost like image classifiers. It does this with a "shuffled frame test" (comparing accuracy on normal versus frame-shuffled clips), which gives a bias ratio of 0.12 on ActivityNet versus 2.95 on Something-Something v2. Is Appearance Free Action Recognition Possible? shows that a video transformer falls from 82.9% to 39.8% when appearance is replaced by flow-warped noise, while humans stay at 89.8%. Synthetic environments have less background variety than reality, so this kind of shortcut is a natural failure mode there.
Some of it supplies the method. We reuse MetaTrans's two-stream design: a transformer with positional embeddings for the temporal stream, the same transformer without them for the static stream, and subtraction of the second from the first. We also reuse its gradient-reversal adversary, a layer that flips gradients so the feature extractor learns to fool a domain classifier. We replace its plain marginal adversary with the class-conditioned discriminator of Conditional Adversarial Domain Adaptation. This addresses the known risk that matching overall feature distributions merges distinct target classes.
Some of it constrains the design. MetaTrans's argument that multi-loss objectives are fragile and costly to tune leads us to add exactly one loss, with no generative reconstruction.
The added loss works as follows. Take a synthetic clip $x$ and a second clip $x'$ that renders the same motion-capture sequence with a different character, texture, terrain and lighting. Let $d(\cdot)$ be the motion embedding after subtraction. The render-consistency loss is a contrastive term. It pulls $d(x)$ and $d(x')$ together. It pushes apart pairs whose appearance is the same but whose motion sequence differs:
$$\mathcal{L}_{rc} = -\log \frac{\exp(\mathrm{sim}(d(x), d(x'))/\tau)}{\sum_{m} \exp(\mathrm{sim}(d(x), d(x_m))/\tau)}$$
Here $\mathrm{sim}$ is cosine similarity and $\tau$ is a temperature. The sum runs over $x'$ and over negatives $x_m$ rendered with the appearance of $x$ but a different motion. The full objective is source classification plus target pseudo-label classification, plus the conditional adversary on $d$, plus $\lambda \mathcal{L}_{rc}$, where $\lambda$ is a single weight.
This constraint is stronger than label-level augmentation. It requires two renders of the same motion instance to have the same embedding, not merely the same class. That is the part an ablation must isolate.
Building the pairs is the main engineering risk. RoCoG-v2 is generated from motion capture through a Unity pipeline with randomized lighting, terrain, characters and viewpoints. We will first try to recover motion-sequence identifiers from the released metadata. If they are absent, we will re-render pairs with the released generator. If the generator is unavailable, we will build pairs with a SMPL body-model rendering pipeline in the style of Synthetic Humans for Action Recognition from Unseen Viewpoints, which re-renders the same body motion under randomized clothing, shape and background.
Viewpoint-varied pairs are kept out of the main loss and tested as a separate variant. Changing the camera changes the image-plane motion, so forcing those embeddings to be identical may be the wrong constraint.
The program also includes an analysis of where the gap lives. We freeze the motion embeddings after render-consistency training and before any adversarial training. On them we train a probe that tries to tell synthetic from real. We train the same kind of probe on the static embeddings. If the motion probe already sits near chance, the adversary is unnecessary and the problem reduces to supervised disentanglement. If the probe succeeds, we have measured a kinematic gap that rendering improvements cannot fix. Either result tells simulator designers something useful.
Baselines
The naive reference point is source-only training of the same backbones on RoCoG-v2's synthetic clips. For it, RoCoG-v2 reports 80.3% accuracy for I3D on ground-to-ground and 34.5% for X3D on air-to-air, against target-only upper bounds of 83.0% and 70.3% (RoCoG-v2). It also reports a roughly 36.2-point gap between source-only and target-only on the compound ground-synthetic-to-aerial-real split.
The strongest published adaptation result on this benchmark is contrastive alignment (CO²A) with X3D on air-to-air, reported as roughly a 29-point improvement over source-only. DANN on the compound split is reported only as a "substantial improvement, though with high variance," with no exact figure. We will rerun both from the released configurations rather than copy numbers.
The strongest method-level competitor is MetaTrans run on RoCoG-v2 without modification. It is the closest design, and it holds the best real-to-real result in the file: 95.4% average accuracy on UCF-HMDB. No reported result found on Lacuna for MetaTrans or TranSVAE on RoCoG-v2, so we will produce both. TranSVAE also reports 93.37% on UCF-HMDB and 52.6% on Epic-Kitchens.
Fairness rules apply to every method. All use the same backbone features, the same synthetic and real clips, the same number of training steps, and the same unlabeled target split for pseudo-labels. Each gets the same hyperparameter budget: an equal number of runs, searched on a held-out labeled slice of the real target that is kept identical for all methods and reported separately. All are run over five seeds.
The comparison also includes a pose-based classifier. It is trained on the simulator's exact skeletons and applied to 2D keypoints that an off-the-shelf pose estimator extracts from real video, with the same compute.
Two ablations isolate the key ingredient. The first, augmentation-only, keeps the same render pairs as extra labeled clips but removes $\mathcal{L}_{rc}$. The second, consistency-without-adversary, drops the conditional adversary.
The proposal counts as beating MetaTrans only if three conditions hold on the two aerial splits. Mean accuracy must improve by more than the across-seed standard deviation of both methods. Seed variance must be no larger. The Move Forward/Move in Reverse confusion rate must be lower.
Why not just…?
Why not just run MetaTrans on RoCoG-v2? This is the most serious threat, and the main comparison is designed to settle it. MetaTrans's order-blind stream can only remove appearance that is constant over the clip. Simulator artifacts such as flickering shadows, rigid character animation and motion-capture timing change over time, so they pass into the temporal stream, where the adversary must then work on entangled features. If MetaTrans matches the proposal on the aerial splits and shows equal forward/reverse confusion, render consistency adds nothing. We would then report that the cheap architectural split is sufficient for synthetic-to-real transfer as well.
Why not just treat the re-renders as more domain randomization? RoCoG-v2 already randomizes terrain, characters, lighting and viewpoint across about 107K clips, and source-only X3D still reaches only 34.5% on air-to-air. More randomization therefore does not close the gap by itself. The sharper version of this objection is that our gains could come simply from having more varied renders rather than from the per-instance constraint. The augmentation-only ablation tests exactly that. If it comes within noise of the full method, the idea reduces to domain randomization and we will say so.
Why not just discard RGB and use optical flow or skeletons? Is Appearance Free Action Recognition Possible? notes that flow keeps silhouettes. TranSVAE reports that RGB-only disentanglement beats RGB-plus-flow adaptation methods on UCF-HMDB. Neither result was obtained on synthetic-to-real data, though. For gesture classes, a skeleton classifier trained on exact synthetic poses is a genuine alternative. Its likely weak point is pose estimation from drone views, but we have no figure for that, so the pose baseline is run in full. Comparing results across splits will show whether it wins on ground views and loses on aerial ones.
Why not just translate synthetic frames to look real? On GTA V to Cityscapes, frame translators lose structure. Content Disentanglement for Semantically Consistent Synthetic-to-Real Domain Adaptation reports 32.4 mIoU for CycleGAN, 34.0 for CUT and 37.4 for MUNIT, against 39.0 for its content-style method and 67.4 for the real-data oracle. Mask-based content losses like those of PixelDA need foreground masks and work frame by frame, without any temporal coherence. We therefore treat per-frame translation as a weaker baseline. We will include one CUT-translated run so the claim is checked on video rather than assumed.
Proposed Experiments
The main experiment covers all four RoCoG-v2 splits: ground-to-ground and air-to-air synthetic-to-real, ground-to-air real-to-real, and the compound ground-synthetic-to-aerial-real split. We report target accuracy as mean and standard deviation over five seeds, along with the full confusion matrix. The real-to-real split has no render pairs, so it serves as a control in which only the architecture and the adversary act.
Three diagnostics test whether any gain comes from motion rather than from better-matched appearance. The first applies the shuffled-frame bias ratio of Rethinking Training Data to the real target test set; a higher ratio means more reliance on temporal order. The second is an appearance-free version of the real test set, built with the noise-warping procedure of Is Appearance Free Action Recognition Possible?. The third is time-reversal sensitivity: the distance between motion embeddings of real clips played forward and backward.
The decisive pattern has four parts. The full method must beat MetaTrans on the compound split, with equal or lower variance. It must beat the augmentation-only ablation. It must raise the bias ratio and appearance-free accuracy, not just raw accuracy. Finally, the frozen-embedding domain probes should locate the residual gap mainly in the motion stream, which would justify keeping the adversary.
A secondary experiment repeats the comparison with SMPL-rendered synthetic clips as the source and NTU RGB+D unseen side views as the target. The real-plus-synthetic augmentation reference there is 69.0% versus 53.6% for real-only (Synthetic Humans for Action Recognition from Unseen Viewpoints). This tests whether render-paired disentanglement carries over to a second simulator.
References
- Unsupervised Video Domain Adaptation: A Disentanglement Perspective
- Return of Frustratingly Easy Unsupervised Video Domain Adaptation
- Synthetic-to-Real Domain Adaptation for Action Recognition: A Dataset and Baseline Performances
- Rethinking Training Data for Mitigating Representation Biases in Action Recognition
- Is Appearance Free Action Recognition Possible?
- Conditional Adversarial Domain Adaptation
- Synthetic Humans for Action Recognition from Unseen Viewpoints
- Content Disentanglement for Semantically Consistent Synthetic-to-Real Domain Adaptation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks