The best skeleton action recognizers let every joint attend to every other joint. They stopped passing messages only along the bones because coordinated movements, such as two hands clapping, involve joints that sit far apart on the anatomical tree. That same all-to-all wiring becomes a liability when a model trained on clean synthetic bodies meets a real depth sensor. One occluded wrist or one jittering ankle sends bad coordinates into every other joint's representation. The domain gap therefore does not stay local: it spreads through the matrix of learned joint-to-joint weights. We propose to treat that matrix, which we call the affinity matrix, as the object to adapt.
The method has two parts. First, a learned corruption module damages clean synthetic skeletons until a critic cannot tell them apart from real sensor skeletons. This removes the cheap "is it noisy?" cue. Second, an adversarial critic aligns the affinity matrices across domains, while the per-joint features are left free to keep the fine kinematic detail that separates similar actions. If the approach works, it explains a known failure and repairs it. Feature-level adversarial alignment distorts skeleton structure, and dense relational models lose their advantage exactly where the sensor is worst.
Core Claim
Our claim is that, for dense-relation skeleton models, most of the synthetic-to-real accuracy loss passes through the affinity matrix. The fix needs two things in a fixed order. Sensor-style noise must first be imitated, so the domain critic cannot take a shortcut. Only then is aligning relations, rather than features, useful. Neither step alone should close the gap. Their combination should keep the large gains that long-range modeling gives on two-hand coordination classes. Without adaptation, we expect those gains to collapse under transfer.
Three findings motivate the claim. Context Aware Graph Convolution for Skeleton-Based Action Recognition (CA-GCN) computes pairwise relevance between all joints. It reports 91.4% on NTU RGB+D cross-view and gains of up to 16.5% on two-hand classes, all evaluated only on clean, in-domain skeletons. Adversarially Masking Synthetic to Mimic Real (ASM) shows that when the source is clean and the target has missing points, a domain discriminator latches onto the noise signature. Its fix is to learn dropout masks on synthetic LiDAR scenes. Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition (TS-Cub) abandons adversarial alignment on skeletons because feature matching distorts kinematic structure, and uses limb-swap and clip-order puzzles instead.
We combine these lessons. From ASM we take the remedy for the noise shortcut, but we move it from static scenes to articulated joints over time. From CA-GCN we take the dense relational backbone. To TS-Cub's objection we answer that the problem lies in what gets aligned, not in adversarial training itself.
Research Program
Motivating diagnostic. We first measure how far corruption propagates. We take a sparse anatomical graph network and a dense CA-GCN-style network, and perturb one joint at a time with occlusion or jitter in the style of ScanNet. We then record how much every other joint's hidden feature changes, and how much the affinity matrix changes. If the dense model's sensitivity to a single joint is spread across the body while the sparse model's stays local, propagation is confirmed. We also expect the affinity matrices of synthetic and real inputs to be more separable than their pooled features. Together these would justify the method. If affinities turn out to be no more domain-revealing than features, the core claim fails early and cheaply.
Corruption imitator. The imitator acts on joint sequences. It outputs per-joint, per-frame drop masks, using the Gumbel-Softmax trick (a way to sample discrete keep/drop decisions while still passing gradients) as ASM does. It also outputs small structured displacements and short-burst temporal dropouts, similar to the deformation and mask controllers of AdaptPoint.
The imitator is trained against an input-level critic that sees raw coordinates from corrupted synthetic sequences and from real sequences. It is also given a perception-guidance term that rewards corruptions the action classifier finds hard. This differs from both precedents. AdaptPoint corrupts rigid objects and ASM corrupts static scenes. Our corruptions must respect the body: extremities are occluded far more often than the torso, and dropouts persist across frames.
Relation-level alignment. The backbone produces an affinity matrix $A(x) \in \mathbb{R}^{V \times V}$ per layer for a sequence $x$ with $V$ joints. A relation critic $D_A$ receives row-normalized affinities and tries to tell corrupted synthetic sequences $M(x_s)$ from real sequences $x_t$. A gradient reversal layer, which flips the critic's gradient on its way back into the backbone, pushes the backbone to make the two indistinguishable:
$$\min_{F,C}\max_{D_A}\; \mathcal{L}_{\text{cls}}\big(C(F(M(x_s))), y_s\big) \;-\; \lambda\, \mathcal{L}_{\text{dom}}\big(D_A(A(M(x_s))),\, D_A(A(x_t))\big)$$
Here $F$ is the backbone, $C$ the classifier, $y_s$ the synthetic labels, $\mathcal{L}_{\text{dom}}$ the critic's binary cross-entropy, and $\lambda$ the alignment weight. Node features receive no adversarial pressure. This is our direct answer to TS-Cub's concern.
The idea of anchoring the adversarial game to structure echoes GCAN. GCAN's graph, however, connects images in a mini-batch. Ours connects joints within one body, so what we align is the body's coordination pattern rather than the arrangement of samples. To handle different skeleton formats, such as 25-joint Kinect and 18-joint OpenPose, the critic reads affinities pooled into five body regions.
Sim-to-real testbed. We build Sim-NTU as follows. We fit SMPL bodies to NTU RGB+D training sequences and smooth them heavily over time, to strip most of the real Kinect artifacts. We then re-render them with BEDLAM-style shape and clothing variation (BEDLAM) through an idealized depth sensor, and extract skeletons from the rendered output. Each synthetic sequence keeps its NTU action label. The target is the real NTU Kinect skeletons of held-out subjects.
This design isolates the sensor and rendering gap while holding motion content roughly fixed. Its weakness is that the fitted motions inherit some real noise, which would make our numbers optimistic. To measure that leakage, we also report results with a deliberately under-smoothed variant.
Baselines
Source-only references. The do-nothing reference is source-only training of two backbones on Sim-NTU, a sparse anatomical graph network and a CA-GCN-style dense network, evaluated directly on real NTU. On the existing cross-dataset benchmarks, the matching reference is the source-only baseline TS-Cub reports: 54.9% for PKU-MMD to NTU RGB+D cross-view, and 22.5% for Kinetics to NTU.
Naive augmentation. The second baseline trains the dense backbone with hand-designed body-aware augmentation: extremity dropout, Gaussian jitter, and burst temporal dropout. Its intensity is tuned on a small labeled real validation split. This is the cheapest serious competitor.
Feature-level adversarial adaptation. The third baseline is standard gradient-reversal alignment of pooled backbone features, as in Adversarial Discriminative Domain Adaptation and Conditional Adversarial Domain Adaptation. It is run both with and without our imitator, so that the choice of what to align can be separated from the noise-shortcut fix.
Strongest existing method. The strongest published method on this question is TS-Cub. It reports 59.2% accuracy on PKU-MMD to NTU RGB+D cross-view and 29.6% on Kinetics to NTU, from Learning from Temporal Spatial Cubism. We will both cite these numbers and re-run TS-Cub on our dense backbone and on Sim-NTU, because its reported numbers use a different backbone.
Ablations. These isolate the key ingredients. We run the imitator alone with no critic, relation alignment alone with no imitator, and the full method with the relation critic replaced by a feature critic.
Fairness and the bar to clear. All methods share the backbone, the data, the unlabeled real pool, a 50-trial hyperparameter budget per method, and the same small labeled real validation split. The success bar has two parts. The full method must beat TS-Cub and the tuned augmentation baseline by more than seed variance, over three seeds, on overall target accuracy. It must also recover at least half of the in-domain dense-over-sparse advantage on the two-hand classes.
Why not just…?
Why not just augment the synthetic data with realistic noise? This is the real threat. AdaptPoint's own finding, that uniform jitter and random dropping fail against structured sensor noise, is evidence about point clouds, not skeletons. Our naive-augmentation baseline and the imitator-only ablation test it directly. If tuned augmentation matches the full method, we will report that relation alignment adds nothing beyond imitating noise, and restate the contribution as the imitator alone.
Why not just use a sparse anatomical graph, which cannot spread noise far? A sparse graph gives up the coordination gains that motivate dense models; CA-GCN reports up to 16.5% on two-hand classes. The per-class analysis will show whether sparsity really transfers better on those classes, or only on actions where long-range coordination matters little.
Why not just use self-supervised puzzles like TS-Cub? TS-Cub's spatial puzzle relies on a fixed five-part partition and never targets sensor corruption. We re-run it on identical backbones and data. A tie on Sim-NTU would mean relation alignment offers no advantage over self-supervision.
Why not just gate each joint by a confidence score, so that bad joints cannot send messages? This is a reasonable architectural fix, but real target skeletons often lack reliable confidence values, and gating does nothing about systematic shifts in the coordination patterns themselves. We include a gated dense backbone as an extra source-only reference and test whether it reduces propagation in the motivating diagnostic.
Proposed Experiments
Benchmarks and metrics. Evaluation covers three settings: Sim-NTU to real NTU (cross-subject), PKU-MMD to NTU RGB+D cross-view (PKU-MMD, NTU RGB+D), and Kinetics to NTU. The metrics are top-1 accuracy overall, accuracy on two-hand classes, and the single-joint propagation sensitivity from the diagnostic.
First decisive check. The relation critic must beat the feature critic when both use the imitator. Otherwise relations are not the right thing to align.
Second decisive check. Relation alignment without the imitator should fail or become unstable, which would reproduce ASM's shortcut finding in skeletons. The full method should succeed.
Third decisive check. After adaptation, the affinity matrices of real targets should sit closer to the synthetic clean affinities, as measured by region-pooled distance. They should also predict accuracy gains per class.
Robustness and limits. We sweep the fraction of the target pool that is unlabeled, and we test transfer to an occlusion-heavy NTU subset. If the full method improves on Kinetics but not Sim-NTU, or the reverse, we will report which gap, rendering or 2D keypoint noise, the method actually addresses.
References
- Context Aware Graph Convolution for Skeleton-Based Action Recognition
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
- Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions
- GCAN: Graph Convolutional Adversarial Network for Unsupervised Domain Adaptation.
- BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion
- Adversarial Discriminative Domain Adaptation
- Conditional Adversarial Domain Adaptation
- PKU-MMD: A Large Scale Benchmark for Skeleton-Based Human Action Understanding
- NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis