Hand gesture recognizers that work from 3D skeletons, meaning the tracked positions of about 22 hand joints over time, are good at broad motions like swipes and circles. They remain poor at fine ones. Pinch and grab look almost identical except for small, coordinated differences between fingertips. Graph neural networks offer a precise way to describe this kind of signal. On the hand graph, "everything moves together" is a low-frequency pattern, and "the thumb tip moves differently from its neighbours" is a high-frequency pattern. Stacking message-passing or attention layers is known to wash out high frequencies, a failure called over-smoothing. Our hypothesis is that fine-gesture confusion is partly over-smoothing in disguise. The information that separates pinch from grab may reach the network's input but be attenuated layer by layer before it reaches the classifier.
The hand graph is tiny, so this hypothesis can be tested exactly rather than through approximations. A 22-node graph can be fully eigendecomposed at negligible cost. That lets us measure, layer by layer, how much class-relevant information each graph-frequency band carries. It also lets us insert a filter, conditioned on the input, that decides per gesture and per layer whether to smooth or to sharpen. The deliverable has two parts. The first is a diagnostic study of whether existing hand-gesture backbones actually lose high-band discriminative information. The second, conditional on the first, is a cheap module that restores it for fine gestures without hurting static and coarse gestures, which benefit from smoothing.
Core Claim
We make two linked claims. The first is empirical. In deep skeleton gesture models such as attention-based dynamic graphs and spatial-temporal graph convolutions (ST-GCN), linear separability of fine gesture pairs drops with depth specifically in the high graph-frequency bands of the joint features. This drop correlates with the per-class errors on finger-versus-whole-hand distinctions. The second is methodological. Fixed high-pass augmentation is the wrong fix, because static postures and noisy coarse gestures want smoothing while fine dynamic gestures want sharpening. An input-conditioned gain on each exact graph-frequency band therefore beats both plain backbones and fixed filterbanks on the fine-grained split, while matching them on the coarse split.
The idea differs from the closest work in specific ways. Skeleton Graph Scattering Networks for 3D Skeleton-based Human Motion Prediction identified over-smoothing in skeleton graph networks, where forecasts collapse to a frozen mean pose. It countered this with trainable band-pass scattering filters over coarse body parts. That work studies regression on full-body motion, uses fixed manual body partitions, and never asks whether over-smoothing causes classification confusions. Complete the Missing Half and Geometric Scattering Attention Networks supply the method family: parallel low-pass and high-pass or band-pass channels, mixed with learned weights. Both use global or node-wise mixing tested only on static citation and web graphs, and both note real overhead from multi-band filtering. On a 22-node hand that overhead disappears. The open question becomes whether the mixing should depend on the gesture, which neither paper addresses. The theoretical license for "sharpen or smooth, chosen per layer" comes from Setting the Record Straight on Transformer Oversmoothing. It shows that smoothing is not inevitable, and that moderate smoothing sometimes beats sharpening on coarse classification (+2.62% vs +1.81% on CIFAR100). That result is exactly why we expect a fixed sharpening filter to hurt static gestures.
Research Program
Spectral basis. Let $L = U \Lambda U^\top$ be the normalized Laplacian of the anatomical hand graph, with $N=22$ joints. Here $U$ holds the eigenvectors (graph Fourier modes) and $\Lambda$ the eigenvalues, sorted from smooth to oscillatory. For a layer's joint features $H_\ell \in \mathbb{R}^{N \times C}$ in a given frame, $\hat H_\ell = U^\top H_\ell$ gives the band coefficients. We group the 22 modes into $B$ bands; we expect $B=4$ to be enough. The lowest mode is essentially the hand's mean, so wrist-centering an input already removes it. This observation becomes important when we discuss baselines. For attention backbones, we repeat the analysis with the eigenbasis of the symmetrized, time-averaged learned attention graph, so the conclusions do not depend on the anatomical graph alone.
Diagnostic. For each layer $\ell$ and band $b$, we train a linear probe to classify gestures using only the band-$b$ component $U_b U_b^\top H_\ell$, pooled over time. We report probe accuracy on confusable pairs, such as grab vs pinch and the finger/hand splits of DHG-28, alongside band energy fractions. We choose probe accuracy over raw similarity distances deliberately. Are we measuring oversmoothing in graph neural networks correctly? questions whether common distance-based over-smoothing measures track anything meaningful, and the quantity we care about is lost discriminability, not lost variance. The hypothesis predicts that high-band probe accuracy on fine pairs peaks early and decays with depth, while low-band accuracy on coarse classes holds steady.
This stage is a gate. If high-band discriminability survives to the last layer in strong backbones, especially those fed bone vectors, then over-smoothing is not the bottleneck. We would publish that negative result rather than build the module.
Module. If the gate passes, we insert after selected layers a spectral gain
$$ H_\ell' = H_\ell + U\,\mathrm{diag}\big(\gamma_\ell(z)\big)\,U^\top H_\ell W_\ell, $$
where $W_\ell$ is a $C \times C$ projection. The term $\gamma_\ell(z) \in [-1, 1]^{N}$ is a per-mode gain, tied within bands and predicted by a small network from a pooled clip summary $z$. Negative gains smooth a band and positive gains sharpen it, mirroring the smooth/sharpen reparameterization of the Transformer analysis but chosen per input. Its cost is two $22 \times 22$ matrix products per frame. We place it after LayerNorm and monitor the sign of the effective update. This follows the observation in Setting the Record Straight on Transformer Oversmoothing that LayerNorm parameters interacting with residuals can silently invert sharpening into smoothing. We deliberately avoid test-time stochastic dropping. Shedding Light on Random Dropping and Oversmoothing shows that DropEdge-style methods act only as training augmentation and destabilize inference if left on. That finding also suggests the neighbourhood dropping of Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition regularizes training rather than preventing test-time homogenization.
Baselines
Naive reference. The do-nothing point is each backbone unmodified, trained with our pipeline and seeds. This separates our gains from reproduction drift.
Strongest existing skeleton methods. On DHG and SHREC'17, the strongest numbers in the file come from Decoupled Representation Learning for Skeleton-Based Gesture Recognition. It uses hand-engineered posture volumes, fingertip trajectory maps, and relative position vectors, and reports 92.54% on DHG-14, 88.86% on DHG-28, 94.88% on SHREC'17-14 and 92.26% on SHREC'17-28. It is the right target because it attacks the same coarse/fine entanglement by explicit kinematic decoupling. Beating it on the 28-class splits, where finger-versus-hand distinctions dominate, is the meaningful bar. We take its reported numbers as the external reference. We also re-run it under our splits if code is obtainable, under the same subject-held-out protocol as the cited papers.
Host backbones. Among graph and attention backbones, Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention is the most natural host. It builds fully connected joint attention, is exactly the dense aggregation most prone to homogenization, and reports 91.9% on DHG-14. Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action-Gesture Recognition is the second host. It is attention-only, uses bone and fast/slow temporal streams, and uses signed (Tanh) attention. No reported result was found on Lacuna for DSTA-Net on DHG or SHREC, so we report only our reproduction. ST-GCN is the third host, since per SHREC 2021 it won the in-the-wild track with Jaccard index 0.8526, detection rate 89.93% and false positive rate 0.066.
Fairness. All hosts, with and without the module, receive identical data, augmentation, epochs, and the same number of hyperparameter-search trials. Results are reported over five seeds.
Ablations. Ablations isolate the key ingredient, which is input-conditioned per-band gain in the exact spectral domain. The first replaces $\gamma_\ell(z)$ with a learned constant, reducing to a filterbank in the spirit of FB-GNN. The second keeps conditioning but gates node-domain channels with matched parameters, a squeeze-and-excitation-style control. The third restricts gains to be non-positive (smoothing only) or non-negative (sharpening only). The fourth adds a parameter-matched plain residual MLP. A win means a statistically clear improvement on DHG-28 and SHREC'17-28, and on the grab/pinch confusion rate, over both the host and the constant-gain ablation, with no loss beyond seed noise on the 14-class splits.
Why not just…?
Why not just feed bone vectors and wrist-centered joints, as existing multi-stream models do? This is the most serious threat. Centering removes the lowest graph mode, and bone vectors are edge differences, which act as a high-pass operator. So these inputs already hand the network high-frequency signal. Our question is whether that signal survives depth, which input design cannot guarantee. The diagnostic runs on bone-stream DSTA-Net specifically. If high-band probe accuracy stays flat across its layers and the module adds nothing on top, we conclude that input engineering suffices and report that.
Why not just use a shallow network so over-smoothing never starts? With 22 joints, two or three layers already cover the graph's diameter, so the receptive-field argument against shallowness is weak here. The evidence against it is indirect. The DG-STA ablations show restricting to local fixed graphs costs 3–5%, but that concerns graph structure rather than depth. We therefore sweep depth from 2 to 8 layers for every host, with and without the module. If the best shallow plain model matches the best gated model, the module is unnecessary.
Why not just rely on signed attention, which can already learn neighbour differences? DSTA-Net found that Tanh beats Softmax precisely because negative weights allow inhibitory relations. In principle this lets attention learn high-pass filters. However, nothing forces it to do so per gesture. The diagnostic on DSTA-Net and a conditioning-off ablation inside it will show whether learned signed attention already recovers what the gate adds.
Why not just add a fixed high-pass channel like FB-GNN? This is our constant-gain ablation. The Transformer smoothing result predicts that it trades static-gesture accuracy for fine-gesture accuracy. We test this prediction by reporting per-category results (static, coarse, fine).
Proposed Experiments
The first experiment is the diagnostic on DHG-14/28 and SHREC'17 for all three hosts at depths 2–8. We report band-wise probe accuracy curves and their correlation with per-class confusion; this result decides whether the module stage proceeds. The second is the module study on the same benchmarks, compared against the baselines and ablations above. The decisive check is a three-way pattern. Gains should concentrate on the 28-class splits and the grab/pinch pair. The learned $\gamma$ should be measurably more negative for static and coarse clips than for fine ones. Restricting the gate to sharpening only should hurt static classes.
The third experiment is a transfer test on the continuous SHREC 2021 stream, inserting the gated ST-GCN into the energy-based detection pipeline. The reference is 0.8526 Jaccard, and static gestures average only 0.73 Jaccard across entrants, so this is where adaptive smoothing should show its value.
A falsifying outcome would be any of the following: high-band discriminability does not decay with depth; constant gains do as well as conditioned ones; or bone-stream inputs alone close the 14-to-28-class gap. Each would be reported as such.
References
- Skeleton Graph Scattering Networks for 3D Skeleton-based Human Motion Prediction
- Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks
- Geometric Scattering Attention Networks
- Setting the Record Straight on Transformer Oversmoothing
- Are we measuring oversmoothing in graph neural networks correctly?
- Shedding Light on Random Dropping and Oversmoothing
- Decoupling GCN with DropGraph Module for Skeleton-Based Action Recognition
- Decoupled Representation Learning for Skeleton-Based Gesture Recognition
- Construct Dynamic Graphs for Hand Gesture Recognition via Spatial-Temporal Attention
- Decoupled Spatial-Temporal Attention Network for Skeleton-Based Action-Gesture Recognition
- SHREC 2021: Track on Skeleton-based Hand Gesture Recognition in the Wild