Action recognition systems that combine video with body pose ("skeletons," the joint coordinates produced by a pose estimator) usually fuse the two at a single spatial scale. A wrist and a torso receive the same treatment, and a well-tracked joint is trusted exactly as much as one the pose estimator has lost. This note proposes that the pose stream should decide two things for every joint in every frame. The first is how large an image neighborhood to read from the video features: a tight window around a fingertip, a wide one around the hips. The second is how much to fall back on whole-frame context when the joint's own location is unreliable. The bet is that a single small gating module, conditioned on joint identity, pose-estimator confidence, and joint speed, can address two failures the literature reports separately. Fine hand and micro-gesture signals get washed out by large body motion. Pose-guided fusion collapses when the pose estimator breaks during flips and fast rotations.
Both failures are documented in the file. BHaRNet splits skeletons into body and hand expert streams and raises hand-intensive NTU accuracy from 86.4% to 93.0%, which shows that scale mixing hurts. FineGym reports that skeleton models degrade badly during gymnastic inversions, while MiGA 2024 and iMiGUE show the reverse on micro-gestures, where pure video overfits to background and skeleton heatmaps are needed as an anchor. No existing model treats "which scale" and "how much to trust the anchor" as one per-joint decision.
Core Claim
The claim is that cross-modal scale selection should be anchored to joints and aware of how reliable each joint is. For joint $j$ at frame $t$, the pose stream supplies a location $p_{jt}$, a detector confidence $c_{jt}$, and a speed $v_{jt}$. A small network $g$ maps these, together with a learned embedding $e_j$ of the joint's identity, to weights over $K$ window radii on the video feature pyramid plus one weight for a global context vector. The video evidence attached to that joint is
$$ z_{jt} = \sum_{k=1}^{K} w_{jt,k}\,\mathrm{Pool}(F_k, p_{jt}, r_k) + w_{jt,0}\, G_t, \qquad w_{jt} = \mathrm{softmax}\big(g(e_j, c_{jt}, v_{jt}, h_{jt})\big). $$
Here $F_k$ is the video feature map at pyramid level $k$, $r_k$ is the window radius at that level, $G_t$ is a pooled whole-frame feature, and $h_{jt}$ is the pose stream's hidden state for the joint. Nothing supervises the scale weights directly; they are learned from the classification loss alone.
Two testable predictions follow. First, hand and finger joints should learn to favor small radii, and the accuracy gain should concentrate on hand-intensive and micro-gesture classes. Second, $w_{jt,0}$ should rise as $c_{jt}$ falls or $v_{jt}$ spikes, and the model should degrade more gracefully than other pose-guided fusion models as pose quality drops.
How This Differs From the Closest Work
Several papers fuse video and skeletons with some notion of scale, and each fixes the part this proposal makes adaptive.
STAR-Transformer builds two fixed token types, global-grid tokens and joint-map tokens, and it relies on clean pre-extracted joints, so it has no notion of joint reliability. RGBSformer fixes one spatial resolution per pathway and reports weakness on hand gestures. The 3D deformable attention model exchanges information between modalities through a single abstract mixer token, which its authors flag as lacking structural grounding. BHaRNet-M guides video with body features only, at one scale. MSA-3D crops hands with a detector at a single scale and fuses modalities only at the decision level.
The closest mechanism is LGMMFusion. It lets LiDAR queries attend to image features with multi-scale deformable attention, but it works on driving scenes, has no articulated-body structure, and has no reliability-driven fallback. This proposal supplies the missing piece in two ways: scale choice is tied to joint identity, and the fallback to global context is tied to the pose estimator's own confidence and speed signals.
Two papers play a different role. The Attention Modality Reweighter motivates the reliability term, since it shows that fusion without adaptive down-weighting is fragile. It reweights whole modalities, whereas this proposal reweights per joint and per scale. ACNet and MSTI-Net supply the evidence that pyramid and dilated multi-scale pooling work within a single modality. They also supply the cost constraint: overhead must stay small, which is why the gate pools a fixed set of windows rather than running a full spatial pyramid module.
Research Program
The model has three parts.
The video stream is a standard pretrained backbone that exposes three or four feature-pyramid levels. The pose stream is a graph network over the joints that outputs a hidden state for each joint. The gating module between them is a two-layer network producing $K+1$ weights per joint per frame. The pooled evidence $z_{jt}$ is added back into the pose stream's joint states before the next graph layer, at two depths, so video evidence can correct the pose stream before any decision is made. This intermediate placement follows RGBSformer's finding that token-level fusion beats score fusion. The extra compute is $K$ windowed pooling operations per joint, which is negligible next to the backbone; GFLOPs and latency will be reported for every model.
On NTU, the skeletons are Kinect 3D joints projected onto the image. On FineGym and iMiGUE, the skeletons come from HRNet, with confidences taken from the estimator's heatmap peaks. Speed is the frame-to-frame displacement normalized by body size.
The analysis program is as important as the accuracy numbers. Three analyses will be run: histograms of learned radius by joint type; plots of $w_{jt,0}$ against detector confidence; and accuracy stratified by per-clip mean pose confidence.
Baselines
The naive reference is late score fusion. The same video backbone and the same graph network are trained separately and their softmax outputs are averaged. Any fusion module that fails to beat this is not worth its complexity. RGBSformer already reports that token-level fusion outperforms score fusion, so this baseline sets a floor rather than a target.
The strongest published competitors differ by benchmark, and the file contains only their reported numbers. On NTU-60 cross-subject, the 3D deformable attention model reports 94.3% accuracy, and STAR-Transformer reports 92.0%. On NTU-120 cross-subject, the best number in the file is BHaRNet-P at 94.3%; that is skeleton-only and is cited as reported. On the hand-intensive NTU subset, BHaRNet reports 93.0%. On FineGym99, the 3D deformable model reports 90.3% mean accuracy and RGBSformer reports 86.7% mean-class accuracy. On iMiGUE, the MiGA 2024 PoseConv3D fusion entry reports 70.25%, against 67.91% for skeleton-only. Beating the published numbers requires matching each paper's protocol: NTU-60 at 12 frames for the deformable model, and its from-scratch setting. Where it is affordable, the 3D deformable model will also be re-run on our backbone, pose inputs, and tuning budget.
The baseline that matters most for the claim is not published as such: joint-query multi-scale deformable cross-attention in the LGMMFusion style. Each joint is a query that attends to learned offsets across all pyramid levels. It uses the same backbone, graph network, injection depths, and hyperparameter search budget of 20 trials per model, with matched GFLOPs within 10%.
Two ablations isolate the key ingredient. The first removes the confidence and speed inputs to the gate, leaving joint identity and hidden state only. The second removes joint identity and keeps reliability. A fixed-scale variant, single radius plus global token, isolates the value of choosing among scales at all.
Beating the deformable baseline means one of two outcomes, each with non-overlapping three-seed intervals. The first is a gain on the hand-intensive NTU subset and iMiGUE. The second is a smaller drop than the baseline in the lowest-confidence FineGym tercile.
Why not just…?
Why not just fuse the two models' scores at the end? Late fusion cannot let video evidence fix a misplaced or blurred joint before the decision is formed. RGBSformer measured this directly on FineGym99 and Kinetics-400, and MSA-3D's late multiplication left inter-modal dependencies on the table. The score-fusion baseline above will confirm this on our setup. If it comes within a point of our model on hand-intensive classes, the fusion module is not justified.
Why not just crop hands with a detector and add a hand-crop video stream? This is cheap and genuinely competitive on hand gestures: MSA-3D's crop added 1.45% on IsoGD. It fixes one scale for one body part, however, and it fails exactly where detectors fail, during fast motion. It will be run as an extra baseline: a YOLO-style hand crop stream added to score fusion. The prediction is that it roughly matches us on iMiGUE but falls behind on FineGym's low-confidence clips. If it matches us on both, the adaptive-scale claim loses.
Why not just use multi-scale deformable attention with joints as queries? This is the real threat. Learned offsets across pyramid levels could implicitly discover both small windows for hands and wider reach when a joint is misplaced. It is therefore the central baseline, with matched compute. The proposal's distinguishing bet is narrow: deformable attention has no input telling it when the anchor itself is untrustworthy, so its offsets stay centered on a wrong location when the pose estimator fails. If the confidence-stratified FineGym analysis shows no gap, the honest conclusion is that deformable attention already suffices. The contribution then reduces to the diagnostic scale analysis, and it will be reported that way.
Why not just reweight whole modalities, as the Attention Modality Reweighter does? A global weight cannot keep trusting the hip while distrusting a blurred hand within the same frame. The test is a modality-level gate variant, one weight per frame, compared on clips with partial joint corruption. These are created by jittering only arm joints with Gaussian noise at three levels.
Proposed Experiments
The benchmarks are NTU RGB+D 60 and 120 in cross-subject and cross-view/setup settings, including the hand-intensive class subset defined by BHaRNet; FineGym99 mean-class accuracy; and iMiGUE top-1 accuracy under the MiGA classification protocol. NTU provides breadth, FineGym tests pose failure, and iMiGUE tests tiny signals inside clutter. Every model is trained with three seeds and reports accuracy, GFLOPs, and latency.
Three decisive checks follow. First, the learned radius distribution must separate finger and hand joints from torso joints. If it does not, the gate is not doing scale selection, and any accuracy gain comes from somewhere else. Second, FineGym accuracy is stratified by mean pose confidence. The full model should lose less than the deformable baseline and the reliability-ablated gate in the bottom tercile while matching them in the top tercile. Third, under controlled arm-joint jitter on NTU, the global weight $w_{jt,0}$ should rise monotonically with noise for the corrupted joints only.
An optional robustness probe applies PGD attacks to the video stream, in the spirit of the Attention Modality Reweighter. The file gives that method's gain only as a relative improvement, so the probe will compare against our re-run of a modality-level reweighting gate rather than a reported figure.
The claim fails if the deformable joint-query baseline matches the full model across all strata.
References
- Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition
- FineGym: A Hierarchical Video Dataset for Fine-Grained Action Understanding
- The 2nd Challenge on Micro-gesture Analysis for Hidden Emotion Understanding (MiGA) 2024: Dataset and Results
- iMiGUE: An Identity-Free Video Dataset for Micro-Gesture Understanding and Emotion Analysis
- STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition
- A Novel Two-Stream Transformer-Based Framework for Multi-Modality Human Action Recognition
- Cross-Modal Learning with 3D Deformable Attention for Action Recognition
- Multi-Scale Attention 3D Convolutional Network for Multimodal Gesture Recognition
- LGMMFusion: A LiDAR-guided multi-modal fusion framework for enhanced 3D object detection
- Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweighter
- Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization
- Multi-Scale Spatial-Temporal Integration Convolutional Tube for Human Action Recognition