A common way to analyze human movement with standard vision models is to turn a skeleton sequence (joint coordinates over time) into a picture. One option is a grid where rows are joints, columns are frames, and color channels store coordinates. Another is a rendered map of Gaussian blobs at the joints, with lines for the limbs. Each choice involves arbitrary decisions: joint order, which joint is the reference origin, and the image format. In large supervised benchmarks these choices cost a few points of accuracy. Tree-based joint ordering beats random ordering by up to 5.5 points on NTU RGB+D 60 (TSRJI). We argue the choices matter far more in the setting where skeletons are most attractive: unsupervised discovery of movement sub-types in clinical cohorts of a few dozen patients. There, nothing tells you which encoding is right, and different encodings may produce different "phenotypes."
The proposal treats a small family of skeleton-to-image encodings as an ensemble of views of each patient. It clusters patients separately under each encoding and combines the results by ensemble clustering, also called consensus clustering. The idea is to merge many partitions by counting how often two samples land in the same group. We then train a small gated cross-attention module to reproduce that consensus. In cross-attention, features from one input decide which parts of another input to read. Here the module fuses the two encoding families. The result is a model that can place new patients into the discovered groups. Its gates also show which encoding and body part carry each distinction. The pretrained image backbones stay frozen throughout, which is how the method respects the small-sample constraint.
Core Claim
We make two claims. The first is diagnostic: in small cohorts, the disagreement between partitions produced by different encodings is larger than the disagreement produced by random clustering seeds. If so, the usual practice of repeating k-means over seeds on one embedding hides the dominant source of instability. The second is constructive: building the consensus over encodings, rather than over seeds, and distilling it into a cross-encoding fusion model should yield sub-groups that agree better with held-out clinical labels, transfer better to a new site, and assign unseen patients more reliably than either seed-ensembles or a single best encoding.
Both claims are falsifiable. If the disagreement between encodings is no larger than the disagreement between seeds, the first claim fails and the project reduces to a negative result. If plain concatenation of encoding features matches the fusion module, the attention component is unnecessary. We would then report the encoding-consensus result alone.
Research Program
Encoding families. The first family is coordinate pseudo-images following Skeleton-to-Image Encoding. That method groups joints into five body parts ordered by anatomical connectivity, maps coordinates to color channels, and resizes the result to a 224×224 image. This lets it reuse an ImageNet-pretrained Vision Transformer refined by masked-autoencoder pretraining on NTU. We vary joint ordering (tree traversal versus part-grouped) and the reference origin. The reference options include the four shoulder and hip anchors of TSRJI. GAVD offers only video, so its poses come from an off-the-shelf 2D estimator. We therefore store (x, y, confidence) in the three channels, which is an adaptation that we test rather than assume.
The second family is Gaussian joint and limb maps in the style of SkeletonGait. These are processed by a gait CNN pretrained on public gait data. The two families describe the body differently. One encodes trajectories on an abstract joint-by-time grid. The other encodes spatial body shape. For that reason we treat them as two modalities rather than as cosmetic variants.
Consensus. For each encoding $e$ and each of several cluster counts, we cluster frozen embeddings with k-means and with a Gaussian mixture. We then form the co-association matrix $$S_{ij}=\frac{1}{M}\sum_{m=1}^{M}\mathbb{1}[\pi_m(i)=\pi_m(j)],$$ where $M$ is the number of base partitions $\pi_m$ and $S_{ij}$ is the fraction in which patients $i$ and $j$ share a cluster. The final groups come from agglomerative clustering on $S$, following the co-occurrence formulation in Clustering aggregation. With fewer than a few hundred patients, the quadratic cost that motivated bipartite spectral consensus is irrelevant.
This design also sidesteps an open problem in DivClust. DivClust controls diversity among clustering heads through a target mutual-information level, and its authors leave the choice of that target open for unseen data. Here diversity comes from encoding choices that someone might plausibly have made, not from a tuned knob. We report the pairwise normalized mutual information among base partitions as a diagnostic rather than an objective.
Distillation into fusion. Body-part tokens from the coordinate-image Vision Transformer query spatial tokens from the heatmap CNN, and the reverse. The output passes through the gate of Kim et al., $\text{out}=g\odot\text{cross}+(1-g)\odot\text{orig}$. The gate exists so that a degraded stream can be ignored. This matters because silhouettes and skeletons fail differently under adverse conditions (OpenGait), and skeleton estimates fail on side views. Only the fusion module, roughly tens of thousands of parameters, is trained. Its loss makes the similarity between fused embeddings $z_i, z_j$ predict consensus co-membership: $$\mathcal{L}=-\sum_{i\neq j}\big[S_{ij}\log\sigma(z_i^\top z_j/\tau)+(1-S_{ij})\log(1-\sigma(z_i^\top z_j/\tau))\big],$$ where $\sigma$ is the logistic function and $\tau$ is a temperature.
We state the obvious limitation openly. On the training patients, the distilled model cannot beat the consensus it imitates. Its claimed value lies in assigning held-out and cross-site patients, and in interpretability through the gate values. All of the decisive tests are designed to measure those two things.
This design differs from its neighbors in specific ways. GaitMA and SkeletonGait++ fuse skeleton maps with silhouettes by cross-attention, but they are trained with identity labels on thousands of subjects. M2M-AlignNet fuses brain scans on small cohorts, but it is supervised by diagnosis labels. In our design, the supervision for the fusion comes from unsupervised agreement across encodings. The encoding papers themselves (S2I, Motion Patches) each commit to a single encoding and evaluate it under supervision.
Baselines
The naive reference is k-means on the frozen embedding of the single default S2I encoding, using the true number of clinical categories. A slightly stronger version averages 20 seeds into a seed-consensus. That second version is the practice we claim is misleading, so it has to be beaten clearly.
The strongest competitor on our specific question is consensus over concatenated features. We concatenate all encoding embeddings, reduce them with PCA, and build the same co-association consensus over seeds and cluster counts. This uses all of our information without attention. A further variant pairs this consensus with nearest-centroid assignment for new patients, which is the inductive counterpart to our fusion model.
We also run DivClust on the concatenated features, with its diversity target swept. This shows whether learned head diversity can substitute for encoding diversity. Its reported gain on CIFAR10 is 81.9% clustering accuracy against 79.0% for its single-head base (DivClust). No result on skeletons or clinical data appears in the file.
As a supervised ceiling, we cite SlowFast on GAVD. It reaches 94% binary abnormality accuracy and 87% when trained on GAVD and tested on CASD (Computer Vision for Clinical Gait Analysis). We found no reported unsupervised clustering result on GAVD on Lacuna. We also train a 5-shot-per-class linear probe on our frozen features as a few-label reference.
Fairness rules apply to every method. All methods share the same pose estimates and frozen backbones. Each method gets the same budget of cluster counts and hyperparameter settings. Selection of anything uses only an unsupervised criterion, namely consensus stability under bootstrap resampling of patients, never the clinical labels.
Two ablations isolate the key ingredients. Encoding-consensus without distillation isolates the effect of encoding diversity. Fusion with the gate fixed at 0.5 isolates the gate.
"Beating" a baseline requires more than a single number. On held-out patients, the method must exceed concatenated-feature consensus with nearest-centroid assignment by a margin outside the bootstrap confidence interval, in adjusted mutual information with the 11 clinician categories. It must also hold that margin on the GAVD→CASD transfer.
Why not just…?
Why not just run k-means many times with different seeds on one good embedding? It is cheap, and with a frozen backbone, DivClust's argument about training cost does not apply. This alternative is exactly what the first experiment tests. We compare the spread of normalized mutual information across seeds within an encoding to the spread across encodings. If the spread across seeds is comparable, this alternative wins and we say so.
Why not just concatenate all the encodings and cluster? This is the real threat. Concatenation gives the clusterer every feature, and attention adds parameters that a small cohort may not support. The case for attention rests on one specific failure mode. Side-view skeletons lose much of their discriminative signal: accuracy falls to 56–66% on side views against 91–97% on front and back views in GAVD. A gate can down-weight a stream for each patient, while concatenation cannot. We therefore stratify every comparison by camera view. If concatenation matches the fusion module on side views, we drop the attention module.
Why not just label a handful of patients and train a few-shot classifier? When clinical categories are known and trusted, that is the better tool. Our premise is that clinicians suspect presentations are more heterogeneous than the existing categories capture. We include the 5-shot probe so readers can see how much the unsupervised groups give up against a few labels on known categories. We also report whether the consensus splits any clinical category into stable sub-groups.
Why not just use a graph network on raw coordinates? Graph models cannot reuse ImageNet weights or handle mismatched joint counts without architectural changes. The image route reaches 70.1% on Toyota Smarthome, with 13 joints, after pretraining on 25-joint NTU data (S2I). Grid maps also beat graph models by 22.9 points on GREW (SkeletonGait). We do not re-litigate this choice.
Proposed Experiments
Controlled check on NTU RGB+D 60. We subsample the data to 20, 50 and 100 subjects in each class subset, and treat action labels as hidden ground truth. Here we measure the variance from encodings against the variance from seeds, along with consensus quality, where reference supervised results exist: 91.0% fine-tuned accuracy for S2I and 73.3% cross-subject accuracy for TSRJI.
Main study on GAVD. We subsample to cohort sizes of 20, 50 and 100 patients, matching the fewer-than-100-subject regime documented by Bria et al.. We hold out 30% of patients for inductive assignment and use CASD as an unseen site. Clinician labels are used only for scoring.
Decisive checks. The project proceeds only if three things hold. The disagreement between encodings must exceed the disagreement between seeds at small cohort sizes. Encoding-consensus must beat seed-consensus. Gated fusion must beat concatenation with nearest-centroid assignment on held-out and side-view patients. We also inspect whether gate values shift toward the heatmap stream on side views. That shift would be the clearest sign that the module learned something concatenation cannot.
References
- Skeleton Image Representation for 3D Action Recognition Based on Tree Structure and Reference Joints
- Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models
- SkeletonGait: Gait Recognition Using Skeleton Maps
- Clustering aggregation
- Ultra-Scalable Spectral Clustering and Ensemble Clustering
- DivClust: Controlling Diversity in Deep Clustering
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- OpenGait: A Comprehensive Benchmark Study for Gait Recognition Toward Better Practicality
- GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition
- 4D Multimodal Co-attention Fusion Network with Latent Contrastive Alignment for Alzheimer's Diagnosis
- Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches
- Computer Vision for Clinical Gait Analysis: A Gait Abnormality Video Dataset
- Towards AI-driven Next Generation Personalized Healthcare and Well-being