Gait recognition systems identify people by how they walk. The strongest current systems combine two views of the same walking body. One is the silhouette, a binary outline of the person cut out from the background. The other is the skeleton, a small set of joint coordinates produced by a pose estimator. The two carry different information. Silhouettes are temporally stable but change when a person puts on a coat or carries a bag. Skeletons are mostly unaffected by clothing but jitter from frame to frame. The value of fusing them lies in what each modality knows that the other does not. Yet the fusion modules in use today work by pulling the two representations together. They use co-attention, mutual learning, and distribution-matching losses. Our hypothesis is that this alignment behaves like the over-smoothing seen in graph neural networks. In those networks, repeated neighbor-averaging makes all node features converge until the distinctions between them are erased. Here the "nodes" are the silhouette and skeleton features of each body part, and what gets erased is the clothing-invariant identity information the skeleton was added to supply.
This matters because the failure would be concentrated in the hardest case, the clothing-change split. It would also be invisible in standard accuracy tables. One reported number in GaitMA is suggestive though not decisive: removing its entire skeleton branch costs only 3.8% Rank-1 on the in-the-wild Gait3D benchmark (GaitMA). We propose to measure cross-modal collapse directly, using the rank-based diagnostics that recent over-smoothing work showed actually track performance. We then test one targeted fix: confine the alignment pressure to a shared component, and leave the cross-modal difference free to carry identity.
Core Claim
The claim has two parts, and both can be falsified. The first is diagnostic. In silhouette–skeleton fusion networks, alignment losses and stacked co-attention reduce the dimensionality of the skeleton-specific part of the representation. That reduced dimensionality predicts clothing-change accuracy better than within-modality measures do. The second is constructive. Treat each body part's two modality features as a two-node graph and borrow the low-pass/high-pass decomposition from filterbank graph networks. Apply the alignment loss only to the shared (low-pass) channel. This recovers clothing-change accuracy, and it does so without the cost of heatmap rendering or 3D mesh fitting.
The diagnostic tool comes from Are we measuring oversmoothing in graph neural networks correctly?. That paper shows two things about trained networks. First, Dirichlet energy, the classic smoothness measure based on differences between neighbors, misses collapse. Second, effective rank tracks accuracy, with an average Pearson correlation of 0.91 across architectures and datasets. Effective rank is the exponentiated entropy of the normalized singular values of a feature matrix, $\text{Erank}(X)=\exp(-\sum_i p_i \ln p_i)$ with $p_i=\sigma_i/\sum_j\sigma_j$ and $\sigma_i$ the singular values of $X$. The same paper warns that high rank alone does not imply useful features, since random noise is full-rank. We therefore pair rank with an identity probe, described below.
The method idea comes from Complete the Missing Half. That paper splits the identity operator into a low-pass aggregation $L_{LP}$ and a high-pass "diversification" operator $I-L_{LP}$, and mixes them with learnable weights. That paper applies the split over the nodes of a data graph. We apply it over the two-node modality graph attached to each body part $k$. Write $s_k$ for the skeleton feature of part $k$, $g_k$ for its silhouette feature, and $\alpha_L,\alpha_H$ for learnable per-part weights computed by a small attention head. The fused feature is
$$ f_k = \alpha_L \cdot \tfrac{1}{2}(s_k+g_k) \;+\; \alpha_H \cdot \tfrac{1}{2}(s_k-g_k), $$
and the cross-modal alignment loss is applied to $\tfrac{1}{2}(s_k+g_k)$ only. On a two-node graph the low and high channels sum exactly to the original features. The high channel is precisely what cross-modal averaging throws away.
Positioning Against Prior Work
The closest existing work brings anti-over-smoothing ideas to skeletons, but always within a single skeleton graph and never for identification. SPGSN uses band-pass graph wavelets to stop 3D motion forecasts from collapsing into a mean pose. SpSt-GCN adds a feature-differencing branch for degree-1 extremity joints in action recognition. Both are unimodal. Both act over joints rather than over modalities. Both target prediction or classification of actions, not identity under clothing change. The fusion papers are the ones that actually do the aligning that we suspect causes collapse. GaitMA aligns heatmap and silhouette features with co-attention, bidirectional mutual learning and a Wasserstein loss (GaitMA). MMGaitFormer restricts co-attention to matching body regions. GaitRef concatenates silhouette and refined-skeleton embeddings. None of them measures what alignment does to the information that is specific to one modality.
Two pieces of prior work constrain the design rather than motivate it. First, GaitGraph2 already feeds bone vectors, meaning differences between connected joints, into its graph network. That is already a first-order high-pass signal over the skeleton graph. For this reason we do not propose yet another within-skeleton high-pass channel. A reviewer could fairly say the bone branch already provides it. Second, the frequency-filter literature reports that diversification channels help a lot on heterophilic graphs but give negligible gains on homophilic ones (Complete the Missing Half). Our proposal only makes sense if the modality graph is "heterophilic" in the relevant sense, meaning the two modalities disagree in ways that matter for identity. The clothing-change split is where that should hold, and the normal-walking split is where it should not. That asymmetry is a built-in prediction.
Independence Promoted Graph Disentangled Networks supplies an optional extra ingredient: an independence penalty between latent channels, based on the Hilbert-Schmidt Independence Criterion. We test it as a variant that decorrelates the low and high channels. It is not the core mechanism.
Research Program
The work has three stages. The first is a diagnostic study on existing fusion designs. We reimplement a GaitMA-style fusion network with a GaitBase-style silhouette encoder and a ResGCN skeleton encoder in the style of GaitGraph2. Coordinate-based skeletons let us avoid heatmap cost. We then vary two knobs: the weight of the Wasserstein alignment loss, and the number of stacked co-attention blocks. At each checkpoint and each fusion depth we compute three quantities. The first is effective rank of skeleton features, silhouette features, and the skeleton residual, which is the skeleton feature after removing its least-squares projection onto silhouette features. The second is a linear identity probe trained on that residual alone. The third is Dirichlet energy over the modality graph, included as the measure we expect to fail. The quantity of interest is the correlation, across checkpoints, between residual rank or probe accuracy and CASIA-B clothing-change (CL) Rank-1. We compare it with the correlation between the same diagnostics and normal-walking (NM) Rank-1.
The second stage builds the decoupled fusion block from the equation above, as a drop-in replacement for the co-attention block. The third stage is a set of ablations. These separate the effect of moving the alignment loss onto the low channel from the effect of the extra mixing weights. They also check whether the gain is just a residual connection under another name. A residual keeps $s_k$, which equals low plus high with equal weights. However, it does not stop an alignment loss from pulling $s_k$ itself toward $g_k$.
Baselines
The do-nothing reference is the silhouette-only encoder. On Gait3D, GaitMA reports GaitBase at 64.6% Rank-1 (GaitMA). Any fusion method must clear this, or the skeleton adds nothing. Next to it we run the skeleton-only ResGCN, GaitGraph2. Its reported result on OUMVLP-Pose is 67.1% Rank-1 with AlphaPose keypoints (GaitGraph2).
The strongest existing fusion competitor is GaitMA. It reports 66.1% Rank-1 on Gait3D, 95.9% average Rank-1 on CASIA-B, and 91.2% on OU-MVLP (GaitMA). It is the strongest competitor because it is exactly the aggressive-alignment design our hypothesis targets. On the clothing-change split specifically, the reference is MMGaitFormer's 94.8% CL Rank-1 on CASIA-B (MMGaitFormer). On GREW it is GaitRef's 53.0% Rank-1 (GaitRef). No result was found on Lacuna for GaitGraph2 on CASIA-B per split, or for GaitRef on Gait3D. Those comparisons will rely on our own reruns.
For fairness, every method is retrained in our codebase. All runs share the same pose estimator outputs, silhouette preprocessing, training schedule, number of epochs and hyperparameter search budget. The search allows a fixed number of trials per method, including our alignment-weight sweep for the baselines. Comparisons are reported against these reruns, with the published numbers shown alongside.
We count it as beating GaitMA if two conditions hold. First, CASIA-B CL Rank-1 and Gait3D Rank-1 must improve beyond seed variance across at least three seeds. Second, NM accuracy must stay within seed variance. A gain only on NM would contradict our mechanism.
Two ablations isolate the key ingredient. The first keeps the decoupled block but applies the alignment loss to the full features. The second keeps the alignment as in GaitMA but adds a plain residual connection.
Why not just…?
Why not just use late score fusion and avoid cross-modal interaction entirely? This is the most serious threat. Summing scores from independent static and dynamic models took the earliest static/dynamic gait work from 83.75% and 87.50% to 97.50% correct classification on 20 subjects (Fusion of Static and Dynamic Body Biometrics for Gait Recognition). Late fusion cannot smooth the modalities together, because they never interact. We run score-level fusion of the two unimodal encoders as a full baseline. If it matches our block on CL, then the correct conclusion is "fuse late," and our diagnostic study becomes the explanation for why. Our block can only justify itself if feature-level interaction helps somewhere. The likely place is Gait3D and GREW, where noisy skeletons benefit from silhouette context, as GaitRef's refinement suggests.
Why not just lower the alignment-loss weight? If alignment is the cause of collapse, shrinking it might suffice. The sweep in stage one answers this directly. Our prediction is that low weight preserves residual rank but loses the NM and Gait3D gains that alignment provides. That would make the decision a trade-off, and decoupling is meant to avoid it. If some weight on the sweep matches our block on every split, we will report that outcome and drop the method claim.
Why not just add a residual connection or use heatmaps as GaitMA does? A residual is algebraically our block with $\alpha_L=\alpha_H$ and the loss on everything. That makes it the second ablation, and it tests whether loss placement, rather than extra capacity, is what matters. Heatmap rendering removes the graph, but it does not remove alignment pressure. GaitMA aligns heatmaps and silhouettes with a Wasserstein loss, so our diagnostic applies to it unchanged. Its rendering cost is a separate drawback that the file notes but does not quantify (GaitMA).
Why not just add high-pass or wavelet channels inside the skeleton network, as SPGSN and SpSt-GCN do? Bone inputs already supply a first-order joint difference. Skeleton graphs also span the whole body within a few layers. Within-skeleton smoothing is therefore a weaker suspect than cross-modal smoothing. We still measure effective rank layer by layer inside the ResGCN. If within-skeleton rank, rather than cross-modal residual rank, turns out to be what predicts CL accuracy, the proposal pivots to the within-skeleton version.
Decisive Checks
All experiments use CASIA-B, split into NM, BG and CL, plus Gait3D, GREW and OUMVLP-Pose. The first decisive check is correlational. Across alignment weights and fusion depths, does skeleton-residual effective rank, or residual probe accuracy, correlate with CL Rank-1 more strongly than with NM Rank-1? And does Dirichlet energy fail to do so, as Are we measuring oversmoothing in graph neural networks correctly? would predict? If residual rank shows no CL-specific relation, the over-smoothing story is wrong.
The second check is interventional. Does the decoupled block raise CL Rank-1 and Gait3D Rank-1 over the rerun GaitMA, while beating both the late-fusion baseline and the best point on the alignment-weight sweep? And does the loss-on-full-features ablation erase the gain? The third check is a stress test. We inject synthetic keypoint jitter and drop joints at increasing rates. If the high channel carries skeleton-specific identity, its learned weight $\alpha_H$ should fall as the skeleton degrades. That would show the gate is responding to modality reliability rather than acting as a constant.
References
- GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition
- Are we measuring oversmoothing in graph neural networks correctly?
- Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks
- Skeleton Graph Scattering Networks for 3D Skeleton-based Human Motion Prediction
- Skeleton-Based Action Recognition with Spatial-Structural Graph Convolution
- Multi-modal Gait Recognition via Effective Spatial-Temporal Feature Fusion
- GaitRef: Gait Recognition with Refined Sequential Skeletons
- Towards a Deeper Understanding of Skeleton-based Gait Recognition
- Independence Promoted Graph Disentangled Networks
- Fusion of Static and Dynamic Body Biometrics for Gait Recognition