Where Skeleton Networks Smooth Away Fine-Grained Actions

Skeleton-based action recognition classifies a video from the 3D positions of about 25 body joints over time. The strongest models are graph neural networks, which update each joint by averaging information from its anatomical neighbours. A long line of graph research warns that repeated averaging causes "over-smoothing": all joints drift toward the same feature vector, and the network loses the local detail that separates similar classes. Several action papers now blame over-smoothing for confusions such as "typing" versus "writing". Their fixes, however, are either restricted to hands and feet or tangled with other architectural changes, so nobody has shown where in a skeleton network the discriminative detail is actually lost.

This proposal measures that directly and then applies the smallest fix the measurement supports. We decompose every layer's features into spatial frequencies on the skeleton graph (smooth whole-body patterns versus joint-to-joint differences) and temporal frequencies (slow drift versus fast oscillation). We then ask which bands still separate the most-confused class pairs at each depth. Our hypothesis is testable and could turn out wrong. We expect that the damaging smoothing in skeleton networks does not happen mainly inside the spatial message passing, where prior work has focused. We expect it instead at the temporal striding and the final global average pooling, which together keep only the whole-body, time-averaged component of each feature channel. If that holds, a readout that keeps fast, joint-local energy should recover fine-grained accuracy more cheaply than redesigning the graph layers.

Why the obvious idea is not enough

The obvious move at this intersection is to add a high-pass "diversification" channel to a skeleton network. Complete the Missing Half splits the identity into a low-pass and a high-pass filter and adds back the half that standard aggregation discards. A simple identity shows that, with scalar mixing weights, this is almost a reparameterized skip connection. Write $A$ for the normalized skeleton adjacency, $H$ for the joint features, and $\alpha_L, \alpha_H$ for the learned mixing weights of the two channels. Then

$$\alpha_L A H + \alpha_H (I - A) H = \alpha_H H + (\alpha_L - \alpha_H) A H,$$

which is a weighted residual plus ordinary aggregation. The same holds per joint if the scalars become per-joint gates. Modern skeleton backbones such as CTR-GCN already carry residual paths, so bolting on a filterbank risks re-deriving something the network already has.

The idea therefore has to be sharper than "add high-pass filtering". We use the identity constructively. The in-layer high-pass fix becomes a control, a gated residual applied to all joints, and the research question becomes where the discriminative signal is lost.

Relation to prior work

Three skeleton papers motivate the gap, and each leaves the key question open.

Skeleton-Based Action Recognition with Spatial-Structural Graph Convolution (SpSt-GCN) diagnoses over-smoothing and "edge-node starvation" of hands and feet. It adds an inverse-distance differentiation branch and FastDTW-derived functional edges for extremity joints only, and reports 87.8% on NTU RGB+D 120 cross-subject. Its baseline is the original ST-GCN at 70.7%, and the new model changes several components at once, so the gain cannot be attributed to reduced smoothing. We test whether differentiation across all joints, or no in-layer change at all, matters once the readout is controlled.

Skeleton Graph Scattering Networks uses band-pass graph wavelets plus a temporal cosine transform to stop motion forecasts from collapsing into mean poses. We borrow its spatial-by-temporal spectral view, but only as a measuring instrument for classification, a task that paper never evaluated.

Graph Contrastive Learning for Skeleton-based Action Recognition (SkeletonGCL) makes learned adjacency matrices class-discriminative, raising CTR-GCN from 89.8% to 90.8% on NTU 60 cross-subject. It regularizes which graph is used, not what averaging over it destroys, so it is complementary to our proposal and we treat it as a competitor that can be stacked with it.

The graph-theory literature both supplies methods and constrains claims. Alleviating Over-Smoothing via Aggregation over Compact Manifolds proves that Euclidean averaging is a contraction and replaces it with aggregation on a hypersphere. Its runtime concern disappears on a 25-node skeleton, so we include it as the principled in-layer remedy. Two papers, The Oversmoothing Fallacy and Graph Neural Networks Do Not Always Oversmooth, argue that deep-GNN failures are often optimization problems rather than homogenization. Relatedly, Are we measuring oversmoothing in graph neural networks correctly? shows that energy-style metrics confuse collapse with benign alignment. We take these as design constraints: we measure recoverable class information with probes rather than raw feature energy, and we track gradient health alongside it.

Finally, Higher-Order Pooling of CNN Features via Kernel Linearization shows that temporal pooling beyond the mean captures co-occurrence structure. This supports our suspicion that time-averaging, not graph averaging, is the weak point.

Research Program

Spectral probe. The first component is a probe applied to a trained CTR-GCN reproduced from its published configuration. At each block, the feature tensor has shape joints × time × channels. We project it onto the eigenvectors of the skeleton graph Laplacian, which give 25 spatial modes ordered from whole-body to joint-contrast. We also project it onto a temporal cosine basis at that block's temporal resolution. Each coefficient is $Z_{k,f} = u_k^\top H c_f$, where $u_k$ is the $k$-th spatial mode, $c_f$ the $f$-th temporal basis vector, and $H$ one channel slice. We group modes into low, middle, and high spatial bands and slow, middle, and fast temporal bands.

Probing on confusable pairs. We take the twenty most-confused class pairs in the baseline's own NTU 120 validation confusion matrix. For each block and each of the nine band cells, we fit a linear probe on those pairs using energy-normalized coefficients. Probe accuracy, not energy, is the collapse measure. This follows the measurement critique above: a band can lose energy harmlessly, but losing separability is real damage. We also run the probes on an untrained copy of the network and log per-block gradient norms. This separates learned homogenization from the optimization pathologies the fallacy papers emphasise.

Competing hypotheses. The probe distinguishes two stories. Under the in-layer story, favoured by SpSt-GCN, high-spatial-band separability of the confusable pairs decays steadily with depth inside the graph blocks. Under the readout story, it survives the graph blocks but vanishes at temporal striding and the final pooling. The final pooling is, by construction, projection onto the lowest spatial mode and the zero temporal frequency of each channel.

Interventions. The probe result decides which intervention we pursue, and we build both families so the comparison is fair.

The in-layer family has three variants. The first is a per-joint gated residual across all joints; by the identity above, this is the filterbank of the missing-half paper and the all-joint generalization of SpSt-GCN's differentiation. The second reimplements SpSt-GCN's edge-only inverse-distance branch. The third is hypersphere aggregation in the style of ACM.

The readout family keeps the global average and adds, from the last block and from the last block before temporal striding, per-joint energy in the fast temporal and high spatial bands. This is second-order information that averaging discards. Alongside it we build its two cheap cousins: plain per-joint flattening, and mean-plus-standard-deviation pooling over time.

Noise check. High-frequency skeleton signal is exactly where Kinect tracking jitter lives. We therefore repeat the analysis with injected joint noise, and with inputs denoised by the latent-state model in the deep state-space noisy-skeleton codebase. This checks that any recovered band carries motion rather than sensor artefacts.

Baselines

Naive reference. The do-nothing point is CTR-GCN retrained in our pipeline, with the same data preprocessing, augmentation, epochs, and three seeds. SkeletonGCL reports it at 89.8% on NTU 60 cross-subject. We use our own reproduction as the reference so that every variant shares its training noise. Every intervention is reported as a difference from this retrained model, with the standard deviation across seeds.

Strongest existing method on NTU 60. This is SkeletonGCL on top of CTR-GCN, reported at 90.8% cross-subject. It is the strongest competitor because it already targets feature ambiguity between classes, costs nothing at inference, and attacks the same symptom we do through a different cause. We run it from its published description on the identical backbone, schedule, and tuning budget. We also run it combined with our winning intervention. Beating it means exceeding its confusable-pair accuracy by more than two seed standard deviations while matching its overall top-1. A gain that persists when the two methods are stacked would additionally show the effects are complementary.

Strongest existing method on NTU 120. SpSt-GCN reports 87.8% cross-subject. CTR-GCN is used by CSRE, which reports gains of up to 1.0% (bone stream) and 0.7% (joint stream) on NTU 120, but no absolute CTR-GCN figure on NTU 120 was found on Lacuna. Because SpSt-GCN changes inputs and backbone together, we compare against its published number as reported. We also compare against our reimplementation of its differentiation branch inside CTR-GCN, which isolates its anti-smoothing ingredient from its other changes.

Key ablation. The ablation that isolates our ingredient is the all-joint gated residual. By the identity in the opening section, it is the scalar filterbank in disguise. If the band-energy readout does not beat it, our readout claim fails.

Controls. PairNorm (PairNorm) and a depth sweep of the backbone serve as the standard over-smoothing controls. All variants get the same hyperparameter search budget of a small fixed grid over learning rate and weight decay.

Why not just…?

Why not just use a shallower network, since the skeleton graph spans only a few hops? This is a serious threat. If the in-layer story is false, depth may barely matter spatially. The depth sweep answers it: we train two, four, and the full ten blocks and run the probes on each. If a shallow model with our readout matches the deep one, we report that finding. It would say that skeleton models over-provision depth, and it would not rescue the in-layer view. The ST-GCN figures in the file (81.5% on NTU 60 cross-subject, 70.7% on NTU 120) show shallow-era models trail badly. That gap, however, mixes depth with many other design differences, so we do not treat it as evidence either way.

Why not just rely on residuals or PairNorm? Residuals are exactly what the opening identity reduces the filterbank to, so we test them as the main control rather than dismiss them. ACM's results show normalization only delays collapse on 60- to 120-layer citation graphs, reaching 66.9% against DGN's 52.6% on Citeseer. Skeleton networks are far shallower, though, so that evidence may not transfer. The probe plus gated-residual comparison settles it on skeleton data.

Why not just train with SkeletonGCL? It improves the graph that gets averaged over, but not what averaging and pooling throw away. If our readout is redundant with it, stacking the two will show no additive gain, and we will say so.

Why not just pool mean and standard deviation over time, or flatten joints before the classifier? This is the most dangerous alternative. Flattening is an orthogonal re-expression of the spatial modes, and mean-plus-std pooling captures much of the temporal energy. If either matches the band-energy readout, we adopt the simpler one. The contribution then becomes the diagnosis of where smoothing hurts, plus that trivial fix.

Why not just bypass graphs with image mappings? Image mappings project the skeleton to a joint-by-time image and use a standard image network, as in Skeletal Movement to Color Map. That approach reaches 78.89% on NTU 60 cross-subject, below graph models. It also scrambles anatomical adjacency, so it does not answer the question here.

Proposed Experiments

Benchmarks and metrics. The main benchmarks are NTU RGB+D 120 (dataset paper) under cross-subject and cross-setup, and NTU RGB+D 60 (dataset paper) under cross-subject and cross-view. Northwestern-UCLA serves as a small-data check. The primary metric is accuracy on the twenty confusable pairs, fixed from the baseline's validation confusion before any intervention is trained. Secondary metrics are overall top-1 and the probe curves.

Decisive check 1: locating the loss. The probe must show where high-band separability of confusable pairs disappears. Under the readout story, it survives the graph blocks and dies at temporal striding or global pooling. Under the in-layer story, it decays through the graph blocks.

Decisive check 2: the matching intervention wins. The intervention matched to the located loss must beat the gated-residual ablation on confusable pairs by more than two seed standard deviations, without lowering overall top-1. If the band-energy readout wins but the probe says loss happens in-layer, or the reverse, our causal account is wrong, and we will report that. If nothing beats the retrained CTR-GCN baseline, the evidence sides with the fallacy papers: skeleton over-smoothing would then be a mislabeled optimization or capacity effect.

Decisive check 3: the gain is motion, not jitter. Any gain must survive noise injection and denoised inputs, so that the recovered band reflects movement rather than sensor artefacts.