Appearance-Guided Diversity: Using RGB to Tell Deep Skeleton Graphs Which Joints Must Stay Distinct

Skeleton-based action recognizers treat the human body as a graph. Each joint is a node, each bone is an edge, and a graph neural network (a model that repeatedly updates each node by averaging in its neighbors' features) learns from 3D joint trajectories. Skeletons are cheap to process and ignore lighting and background, but they cannot see what the hands are holding. Stacking more averaging layers also produces over-smoothing: joint features blur toward one shared vector, so the wrist starts to look like the hip. Existing fixes for over-smoothing restore diversity generically. They renormalize feature spread, add high-pass filters, or push non-neighbors apart. As Are we measuring oversmoothing in graph neural networks correctly? points out, diversity alone is not enough, because random noise has full rank and no meaning.

We propose to let a second modality decide which diversity is worth keeping. During training only, a frozen RGB video network gives each joint an appearance descriptor, pooled from the image region around that joint. We then ask the skeleton network's deep layers to reproduce the RGB model's soft pattern of joint-to-joint similarity within each clip. In that pattern, a hand gripping a phone is unlike the elbow, and the two hands in a clap resemble each other. At test time the model is skeleton-only and costs nothing extra. The bet is that cross-modal relational distillation acts both as a remedy for over-smoothing and as a way past the information ceiling of skeleton data. It should keep joint representations distinct exactly where the appearance evidence says they should be distinct.

Core Claim

Over-smoothing in skeleton networks and the context blindness of skeleton data are one problem seen from two sides. The deep layers lose joint-level distinctions, and the skeleton never had object and contact distinctions to begin with. A per-clip, joint-by-joint similarity target taken from RGB fixes both, as long as it is matched softly and relationally rather than feature-to-feature.

Concretely, let $H^{(l)} \in \mathbb{R}^{J \times d}$ be the student's features for the $J$ joints of one clip at layer $l$. Let $A \in \mathbb{R}^{J \times d'}$ be the teacher's per-joint appearance features. We define row-softmax similarity distributions $P^{(l)}_s(i,\cdot) = \mathrm{softmax}_j(\cos(h_i, h_j)/\tau)$ and $P_t(i,\cdot) = \mathrm{softmax}_j(\cos(a_i, a_j)/\tau)$, where $\tau$ is a temperature and $i, j$ index joints. The added loss is

$$ \mathcal{L}_{\text{AGD}} = \sum_{l \in \mathcal{S}} \frac{1}{J}\sum_{i=1}^{J} \max\big(0,\ \mathrm{KL}(P_t(i,\cdot)\,\|\,P^{(l)}_s(i,\cdot)) - m\big), $$

where $\mathcal{S}$ is a set of intermediate layers and $m$ is a slack margin. The loss is added to the usual classification loss.

A fully over-smoothed layer makes every row of $P_s$ uniform, so it is penalized directly. A layer that is diverse in arbitrary directions is penalized too, because its diversity does not match the appearance-derived pattern. The design reuses and recombines three existing ideas:

  • relational distribution distillation from I²MD, which distills sample-to-sample similarity distributions between skeleton streams (joints, bones, motion);
  • the soft-margin slack of Cross-Modal Distillation For Widely Differing Modalities, which showed that hard point-to-point matching across dissimilar modalities overfits modality noise;
  • the ego-anchored contrast of Don't Lose Yourself (RedNnD), which fights over-smoothing in multimodal recommendation graphs by anchoring aggregated nodes to their own layer-0 features.

Our design differs from each in a specific way. Unlike I²MD, the target is joint-to-joint within a clip, comes from a different sensor (RGB), and is applied at depth as an anti-smoothing signal. I²MD's target is sample-to-sample and stays within skeleton-derived views. Unlike RedNnD, the anchor comes from another modality rather than the node's own input, and the model is skeleton-only at inference. Unlike GS-MCC, which separates low- and high-frequency signals in a multimodal conversation graph but needs all modalities at test time and still collapses when deeper, we need no extra modality at inference.

This also addresses a tension in the research file. Alignment objectives such as CoCLR and ViCC pull modalities together. Diversification methods such as the filterbank GNN of Luan et al. and the inverse-distance operator of SpSt-GCN push graph neighbors apart. Here we align relational structure across modalities while that structure itself enforces diversity inside the graph.

Research Program

The build has three parts.

Teacher signal. We freeze an RGB video backbone and project each joint into the synchronized color frame. We pool backbone features in a small window around each projected joint and average over time. This gives a $J \times d'$ appearance matrix per clip. We will also test a variant whose window grows with joint confidence, to handle occlusion.

Student. The student is a standard spatial-temporal skeleton graph network. We train it at several depths, so we can test whether the loss changes how accuracy behaves as depth grows and not only the accuracy peak.

Diagnostics. At every layer we log the effective rank of each clip's joint feature matrix, using the entropy-of-singular-values measure from Are we measuring oversmoothing in graph neural networks correctly?. We also log a "useful rank" proxy: the agreement between the student's and teacher's joint-similarity matrices on held-out clips. The research file notes that no multimodal graph work has measured rank collapse. Here rank is a primary outcome, not decoration.

The analysis tests two predictions that would distinguish our mechanism from generic regularization.

  1. The gain should concentrate on human-object interaction classes, where Sym-GNN and RGBSformer both say skeletons lack context. Per-class gain should therefore correlate with the per-class accuracy gap between an RGB model and a skeleton model.
  2. The rank preserved at deep layers should sit disproportionately on the low-degree extremity joints (hands and feet). These are the nodes that SpSt-GCN identified as both most discriminative and most starved of information.

We will also measure gradient rank across layers. Backward Oversmoothing argues that deep graph networks are hard to train partly because gradients collapse, and a loss injected at intermediate layers may act there as well.

Baselines

Naive reference. The same skeleton backbone at the same depths, trained with cross-entropy alone and with standard residual connections and normalization. This shows how badly over-smoothing actually hurts at each depth on this data.

Strongest published skeleton-only numbers. - On NTU-60, Sym-GNN reports 90.1% top-1 cross-subject and 96.4% cross-view. The skeleton stream of RGBSformer reports 91.1% top-1; the protocol is not stated in our file. - On NTU-120, SpSt-GCN reports 87.8% cross-subject, and RGBSformer's skeleton stream reports 85.7%.

We cite these as reference points. Because backbones differ, the comparison that counts is against our own reruns under identical data splits, epochs, augmentation and hyperparameter search budget.

Strongest generic anti-oversmoothing competitors. These are PairNorm, which keeps total pairwise feature spread constant at every layer, and Deeper-GXX from Tackling Oversmoothing of GNNs with Contrastive Learning, which uses decaying residuals plus a contrastive loss that pulls graph neighbors together and pushes others apart. Deeper-GXX reports 80.59% on Cora at 60 layers versus 29.62% for vanilla GCN, but no reported result was found on Lacuna for either method on skeleton action recognition. We will port both to the same backbone with matched tuning budgets. They are the real threat, because they also restore joint diversity, just without appearance guidance.

Strongest cross-modal distillation competitor. Sample-level logit distillation from the same frozen RGB teacher into the skeleton student, using weighted KL divergence as in Multimodal Distillation for Egocentric Action Recognition. That paper reports a 7.3% top-1 gain over an RGB baseline on Something-Else, in a different student modality. The teacher and compute budget are identical to ours. Only the form of the transferred signal differs.

Ablations. Two ablations isolate the key ingredient. The first replaces the RGB per-joint target with the student's own layer-0 joint features, a RedNnD-style self-anchor. The second replaces it with RGB features pooled at randomly permuted joint locations. A third variant replaces the soft-margin KL with $L_2$ feature regression.

What counts as beating the baselines. The method beats the field if it improves over the best of PairNorm, Deeper-GXX and logit distillation at standard depth by a margin larger than seed variance across three seeds. It must also hold accuracy flat as depth increases while those methods decline, and it must beat the self-anchor and permuted-location ablations.

Why not just…?

Why not just keep the skeleton network shallow? If shallow networks capture everything, depth experiments are moot. Evidence from motion forecasting in Skeleton Graph Scattering Networks suggests long-range limb coordination needs larger receptive fields. Still, the honest answer is that our main claim does not rest on depth. Appearance-guided relations should also inject object context a shallow network cannot get from skeletons at all. If gains appear only at depths no one would deploy, we will report the method as a depth-enabling regularizer of limited practical value.

Why not just use PairNorm or an effective-rank penalty? These restore spread without saying which spread matters. The rank paper itself warns that high rank is necessary but not sufficient. The decisive comparison is per-class: if PairNorm matches us on interaction classes, appearance guidance adds nothing and we would drop the cross-modal claim. We expect it not to, because spread alone cannot tell the model what object is in the hand.

Why not just distill the RGB teacher's logits? Logit distillation transfers class-level knowledge and may capture much of the object-context benefit. This is the strongest threat. It does nothing, however, to stop the deep joint features from collapsing. The experiment that settles this is the depth sweep combined with per-layer effective rank. If logit distillation matches us at every depth and leaves rank equally healthy, the node-level formulation is unnecessary. We will also test the two together, since they are complementary in principle.

Why not just anchor each joint to its own input features? This is RedNnD's move, and it needs no RGB. If it matches the RGB target, the result is cheaper and we should adopt it. The self-anchor ablation exists to decide exactly this. Our prediction is that it helps with over-smoothing but not with interaction classes, since layer-0 skeleton features contain no object information.

Proposed Experiments

Main experiments. These run on NTU-60 (cross-subject and cross-view) and NTU-120 (cross-subject and cross-setup), which provide synchronized RGB and 25-joint skeletons. We report top-1 accuracy across a depth sweep and per-layer effective rank of the joint features. We also report per-class gains split by the dataset's interaction versus body-only categories and by the per-class RGB-minus-skeleton gap.

Low-label regime. We test whether the appearance target substitutes for labels, using 1% labeled data in a linear-evaluation setting. The reference here is I²MD: 83.4% cross-subject and 88.0% cross-view on NTU-60, and 73.1% cross-subject on NTU-120, under linear evaluation.

Occlusion stress test. We drop out extremity joints at test time, probing the missing-feature setting where PairNorm found deep propagation essential (74.5% versus 42.3% on Cora with missing features).

Decisive checks. Three checks determine whether the mechanism is real, independent of headline accuracy.

  1. Accuracy gains should correlate positively with the RGB-skeleton per-class gap across classes.
  2. The permuted-location control should lose most of the gain. That would show the benefit comes from joint-specific appearance, not from any extra regularizing target.
  3. Effective rank under our loss should track accuracy across depths more closely than under PairNorm. That would show the preserved dimensions are useful ones, and would directly answer the "high rank is not enough" objection.