Factoring Out Style to Help Rare Actions: Explicit Nuisance Separation for Long-Tailed and Anomalous Human Motion

When a recognizer sees only a dozen examples of an action, it cannot learn from data that a fast, short-armed "wave" and a slow, long-armed "wave" are the same thing. Those differences in speed, body size and camera angle are the main source of variation within a class. Deep skeleton networks learn to ignore them by seeing many examples, and the rare classes are exactly where those examples are missing. We propose to remove these differences by construction. Each skeleton sequence is split into two parts. The shape part is invariant to rotation, body scale and execution speed, built from classical geometric tools. The nuisance part is a small explicit record of the scale, orientation, speed profile and handedness that were removed. A model then learns from the shape part plus a deliberately limited view of the nuisance part.

The claim is that hand-built invariance is worth the most where data are scarcest. If that holds, the gain from factorization should grow steadily as class frequency falls. It should help tail classes in long-tailed recognition, and it should cut false alarms in skeleton anomaly detection, where unusual but normal executions are mistaken for anomalies. The same representation is tested in both regimes. Feature engineering supplies the mechanism, intra-class variation is what gets factored, long-tailed imbalance is the main testbed, and rare-event detection is a second stress test.

Core Claim

The long-tailed skeleton literature treats tail failure as a sampling or loss problem. Shap-Mix uses Shapley-guided body-part mixing in a graph network. Balanced Representation Learning (BRL) combines partial mixup, temporal reversal and a two-stage effective-count loss. Both create more tail-like data, but neither changes what the network must learn to be invariant to.

The feature-engineering line attacks intra-class variation directly. Szczapa et al. map skeletons to Gram matrices on the manifold of positive semi-definite matrices, smooth the trajectories with Bézier curves, and classify with an alignment-kernel SVM. Shan et al. apply slow feature analysis to multi-order joint geometry. That work was only ever evaluated on small balanced benchmarks, and a kernel SVM scales quadratically with dataset size.

Our hypothesis sits in between. Explicit invariance is a data substitute, so its marginal value is inversely related to class sample count. A deep backbone plus a rebalancing loss still needs it for the tail and does not need it for the head. The proposal does not replace graph networks with SVMs. It also does not claim a new rebalancing loss. It feeds a factorized input to the same backbones and losses the strongest methods use, and it predicts where the gain appears.

A second, testable consequence concerns class prototypes. Yao et al. report that averaging skeletal executions corrupts the class template, because a high-elbow wave and a low-elbow wave blur together; prototype learning for micro-gestures describes the same failure. We predict that much of this failure comes from unaligned nuisance factors. In the canonical shape space, a nearest-class-mean classifier for tail classes should then recover. Recovery would make an extremely cheap tail classifier viable.

Research Program

The representation is built in three steps. First, the joint coordinates at each frame are centered to give $Z_t$. We take the Gram matrix $G_t = Z_t Z_t^\top / \mathrm{tr}(Z_t Z_t^\top)$, which is unchanged by any rotation of the body or camera and by uniform body scale. Second, the Gram trajectory is smoothed, following Szczapa et al.'s Bézier fitting, so that pose-tracker jitter does not inflate path length. Third, the smoothed trajectory is re-timed by its own progress:

$$\tau(t) = \frac{\sum_{k \le t} d(G_k, G_{k-1})}{\sum_{k \le T} d(G_k, G_{k-1})},$$

Here $d$ is a distance between consecutive Gram matrices and $T$ is the sequence length. Resampling at uniform $\tau$ gives a shape sequence $s(\tau)$ in which a fast and a slow execution of the same motion coincide.

Everything removed is kept in a nuisance vector $\nu$. It contains the trace (body scale), the rotation recovered by Procrustes alignment, the speed profile $dt/d\tau$, and a chirality sign. Gram matrices cannot tell a left-handed action from its mirror image, so the chirality sign is computed from the sign of a determinant of torso and limb vectors.

The design variable that matters is how the model may use $\nu$. Discarding it entirely risks the invariance-versus-discrimination trade-off noted in the action literature: "sit down" and "fall" may differ mainly in speed. We therefore study three settings. In the first, $\nu$ is dropped. In the second, $\nu$ is concatenated as a low-dimensional side input. In the third, $\nu$ passes through a bottleneck, so the classifier can read coarse speed but cannot memorize style.

For long-tailed recognition, the factorized input $(s, \nu)$ replaces raw joints in the GCN backbone and training recipe used by BRL and Shap-Mix. A graph convolutional network (GCN) is a network that passes information along the skeleton's bone graph. The loss and the augmentation stay unchanged. We also test a cheap variant: tail classes with fewer than 20 samples are classified by nearest class mean in the time-aligned shape space, and the network handles the rest.

For anomaly detection, the same factorization feeds the one-class objective of COSKAD. COSKAD pulls normal skeleton embeddings toward a centre in Euclidean, spherical or hyperbolic space. The anomaly score is split into a shape deviation and a nuisance deviation. The nuisance deviation is measured conditionally on the nearest normal shape cluster, so a fast walk is only alarming if fast walks never occur among normal walks.

Two papers shape this program. DC-Prophet filters normal bulk with a one-class SVM before a supervised stage; we reuse that decoupling as an optional second stage. UBnormal argues that pure one-class models hit a ceiling, which constrains how far a one-class shape score alone can go.

Baselines

The naive reference point is the backbone trained on raw joints with plain cross-entropy on the long-tailed split. On LT-NTU 120, Shap-Mix reports this as 64.2% overall and 54.7% on classes with fewer than 20 samples.

The strongest recognition competitors are Shap-Mix and BRL. They are strongest because they target the same failure, tail collapse in skeletons, with skeleton-aware augmentation and rebalancing. Shap-Mix reports 73.0% overall and 69.7% few-shot accuracy on LT-NTU 120 (the earlier GLMC method reaches 71.5%). BRL reports 81.8% cross-subject accuracy and 67.8% few-shot accuracy on LT-NTU 60, against 77.3% for the prior best. For Shap-Mix on LT-NTU 60 and BRL on LT-NTU 120, no reported result was found on Lacuna, and the same holds for both methods on Kinetics Skeleton 400. We will rerun both with released or reimplemented code on identical splits. Compute, epochs and hyperparameter search budget will match, and the search budget given to our variant will not exceed theirs. Beating them means a statistically reliable gain in few-shot-class accuracy across three seeds without losing more than one point on head classes. Adding factorization on top of BRL or Shap-Mix and gaining also counts.

The ablation that isolates the key ingredient is the "augmentation-equivalent" baseline. It is the same backbone with random rotation, uniform scaling and random time-warping drawn from the same transformation group our canonicalization removes. A second ablation uses canonicalization with $\nu$ dropped, to measure the cost of full invariance.

For anomaly detection, the competitors are COSKAD and GiCiSAD. GiCiSAD forecasts diverse plausible normal futures with a graph-conditioned diffusion model and adds a body-part jigsaw task. The reported numbers are COSKAD at 65.5 AUC on HR-UBnormal, 87.8 on HR-Avenue and 77.1 on HR-STC, and GiCiSAD at 89.6 AUROC on HR-Avenue and 78.0 on HR-STC. No GiCiSAD result on HR-UBnormal was found on Lacuna. The 93.0% Avenue figure in UBnormal comes from a pixel-based supervised setting and is not comparable to the human-related skeleton subsets. Beating COSKAD means the factorized COSKAD exceeds its own raw-input version under the same pose tracker, epochs and center-update schedule. We will also report the false-positive rate at fixed recall, since the file stresses that tiny false-alarm rates become floods at $10^4{:}1$ imbalance.

Why not just…?

Why not just augment the graph network with random rotation, scaling and time-warping? This is the most serious threat, because augmentation encodes the same invariances without changing the input. The argument against it is about sample efficiency. Augmentation makes the network learn invariance from the samples it is shown, and tail classes contribute few of those samples. There is also the classifier-bias gradient described for rare classes, which keeps pushing tail decisions toward head classes. Canonicalization delivers the invariance with zero samples. We do not consider this settled. The augmentation-equivalent ablation decides it directly. If augmentation matches factorization on the few-shot classes, the central claim is wrong and we will report that.

Why not just rebalance, with oversampling or a class-weighted loss? Few-Shot Learning With Class Imbalance shows that random oversampling recovers 5 to 20 points on imbalanced tasks, so rebalancing is not weak. Rebalancing reweights the examples a class already has; it does not make a slow execution resemble a fast one. We therefore treat rebalancing as complementary. Every factorized variant is trained with BRL's own rebalanced loss. The test is whether the gains add up. If factorization adds nothing on top of rebalancing, the claim fails.

Why not just use Szczapa-style manifold features with an SVM, as the feature-engineering literature suggests? Those results, such as 97.99% on UTKinect and 92.44% on UAV-Gesture, come from small balanced sets. The alignment kernel scales quadratically, which is impractical at NTU scale. We expect a pure SVM to lose on head classes, where learned features have ample data. We run it anyway on LT-NTU 60 with an approximate-kernel variant as a reference. Its per-class curve against class frequency is itself evidence about where engineered features stop helping.

Proposed Experiments and Decisive Checks

The first decisive check is a frequency curve. On LT-NTU 60 and 120, we plot per-class accuracy gain (factorized minus raw, same loss) against log training count. The hypothesis predicts a negative slope with a crossover near the head. A flat curve refutes it.

The second check is a controlled dose experiment. On a balanced subset of NTU 60, we subsample every class to $n$ examples, for $n$ from 5 to 200, and track the gap between the factorized model, the raw model and the augmentation-equivalent model. This removes the confound that tail classes may simply be intrinsically harder.

The third check is a nuisance-shift test. Tail classes are trained only on slow executions or only on one subject-height band, then tested on the complement. Here factorization should help most, and augmentation can only partly help.

The fourth check tests the prototype claim. On MSR Action3D and the tail of LT-NTU, we compare nearest-class-mean accuracy in raw joint space, in Gram space without re-timing, and in the full shape space.

The fifth check is a false-alarm attribution on HR-STC and HR-UBnormal. For COSKAD's top-scoring false positives, we measure what fraction are extreme in nuisance coordinates but typical in shape. The hypothesis needs that fraction to be large, and it needs the conditional nuisance score to move those cases below threshold without losing true anomalies.

Two failure modes will be reported explicitly. First, re-timing by progress is unstable for very short events, so results will be stratified by sequence length. Second, upstream pose errors from AlphaPose or PoseFlow propagate into the Gram matrices, so we will measure robustness under synthetic joint dropout.