Touchless interfaces such as wall displays, AR headsets and rehabilitation games have to answer two questions on every frame. Is someone issuing a command right now, and which hand is doing it? Today these questions are answered by stacked components. A hand or landmark tracker runs first, a sliding window over recent frames comes next, a threshold decides when a gesture has started, and a separate classifier runs last. Each stage fails in its own way. Landmark trackers lose the hand when it closes or blurs. Sliding windows have fixed lengths that do not fit fast and slow performers equally well. Thresholds fire on resting or fidgeting hands.
We propose to collapse this stack into one network that treats each gesture as a point in space-time. An anchor-free detector is a model that, instead of scoring thousands of predefined candidate boxes, predicts a heatmap whose peaks mark object centres and reads the object's size off the feature at each peak. We extend that idea from the image plane to a video volume. The network outputs a heatmap over position and time. Each peak marks where a hand is and the moment a gesture is at its core. From the features at that peak it reads off the gesture class, the gesture's start and end, the hand's box, and a small code describing how the fingers move locally. One forward pass over a streaming feature buffer replaces the tracker, the window, the gate and the classifier.
Core Claim
Detecting gestures as space-time centre points, with a per-peak local motion code, should match multi-stage pipelines on standard continuous-gesture benchmarks at a fraction of the compute. It should also degrade much more gracefully than landmark-based pipelines when hands are far away, blurred, or partly closed. Those are the conditions that HaGRID argues favour box-level representations over skeletons. The new element is not anchor-free temporal localization in general, which exists in the broader video action-detection literature but not in this file. The new element is the joint use of spatial hand peaks and temporal gesture peaks, together with a local motion code at each peak in place of a skeleton. That code is what should let a box-level detector tell "pinch" from "grab."
We do not claim the model will beat skeleton pipelines everywhere. At close range with clean tracking, a landmark-plus-graph model may match or beat it. The claim is about where the curves cross, and about efficiency.
How It Differs From the Closest Work
Four papers come closest, and each fixes only part of the problem.
HaGRID builds a 554,800-image static-gesture dataset with hand boxes and a "no gesture" class. It argues that boxes survive 0.5–4 m distances and harsh lighting better than landmarks. It then builds dynamic gestures by queuing static predictions. We keep its argument for box-level robustness and its data for spatial pretraining. We replace the heuristic queue with learned temporal peaks.
Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks gates a heavy 32-frame ResNeXt-101 classifier with a light 8-frame ResNet-10 detector over sliding windows. It is our main efficiency competitor. We remove both the windows and the gate.
SHREC 2021 shows that graph networks over Leap Motion skeletons plus an energy-based segment detector win at detecting gestures embedded in ongoing movement. However, the inputs are pre-extracted skeletons from dedicated hardware, and fine finger gestures remain the weak class. It motivates our focus on fine dynamics and our false-positive metrics.
On-device Real-time Custom Hand Gesture Recognition runs MediaPipe landmarks at about 60 FPS on a phone. It reports that its classifiers latch onto static hand shape and ignore swipes and waves. This motivates our second claim: a representation built on motion rather than pose should not have that bias.
Three papers supply the method. CenterNet-style peak detection, which needs no non-maximum suppression and suits low-power hardware as Spiking CenterNet shows, gives the decoding scheme. CenterMask predicts a coarse 32×32 local shape vector at each centre and multiplies it with a global saliency map. That gives the template for our local motion code, which we predict as a coarse local flow patch rather than a mask. From MASA we borrow the idea of supervising on motion residuals rather than static pose.
One paper constrains the design. The UAV centre-keypoint tracker reports that centre heads collapse when two targets share a centroid. Two-handed and crossing-hand gestures will trigger exactly this, so the design must handle it explicitly.
Research Program
The model has three parts.
A streaming backbone computes per-frame 2D features at stride 4 using a DLA-34-scale network pretrained on HaGRID boxes. A lightweight causal temporal module, a few dilated temporal convolutions over a rolling buffer, sits on top of it. The causal design means each frame is processed once rather than re-processed in overlapping windows.
The head predicts a space-time heatmap with one channel per gesture class. The training target is a Gaussian placed at each gesture's hand centre and temporal midpoint:
$$Y_{xytc} = \exp\left(-\frac{(x-p_x)^2+(y-p_y)^2}{2\sigma_s^2} - \frac{(t-\tau_c)^2}{2\sigma_t^2}\right)$$
Here $(p_x, p_y)$ is the hand-box centre on the feature grid and $\tau_c$ is the gesture's temporal centre. The spatial width $\sigma_s$ scales with box size and the temporal width $\sigma_t$ scales with gesture duration. The loss is CenterNet's focal loss.
At each peak, parallel heads regress four quantities. The first is the hand box. The second is the distances back to the gesture's start and forward to its end, so duration is predicted rather than fixed by a window. The third is a 2D handedness and identity embedding. The fourth is the local motion code, a small grid of predicted displacement vectors inside the hand box. We supervise this code with optical flow computed offline on training video, restricted to the box.
The motion code is the ingredient that should carry fine-gesture discrimination. It also gives the network a training signal that cannot be satisfied by memorising static shape.
To handle centroid collision, we use two measures. Two-hand gestures get a dedicated "bimanual" centre at the midpoint of both hands, with two box offsets. Same-location collisions between different gestures are separated in time, because their temporal centres differ.
To trigger early in online use, we fire as soon as the causal heatmap value at a candidate peak crosses a confidence level, and we never wait for the end. This mirrors the gesture-"nucleus" early triggering of the two-stage CNN work. We report how many frames before the true temporal centre the trigger fires.
Baselines
The naive reference point is a HaGRID-style pipeline: a per-frame anchor-free hand detector-classifier trained on the same HaGRID data, followed by a majority-vote queue over the last $k$ frames, with $k$ tuned on validation data. If our model cannot clearly beat this on dynamic classes, the temporal machinery is not earning its cost. The file reports 97.5% mAP for the static detector in HaGRID. There is no reported result on Lacuna for queue-based dynamic recognition on any continuous benchmark.
The strongest efficiency competitor is the gated detector-classifier of Köpüklü et al.. It reports 91.04% Levenshtein accuracy on EgoGesture in online mode, a metric that counts insertions, deletions and substitutions of single-time activations. We will retrain it with its published configuration and with a FLOP-matched smaller classifier. Both versions use the same EgoGesture splits and RGB input as our model, so the comparison does not mix in the depth results the paper also reports. Counting as a win means matching 91.04% within one point at substantially lower average compute per frame, or exceeding it at equal compute. For nvGesture, the file gives only an offline depth accuracy of 83.82% and no reported result on Lacuna for online nvGesture performance, so we will establish that number ourselves.
The strongest representational competitor is MediaPipe landmarks feeding a spatial-temporal graph convolution network with an energy-based segment detector. This is the SHREC 2021 winning recipe, which reports a Jaccard index of 0.8526 and a false-positive rate of 0.066 on SHREC 2021. Because SHREC provides only Leap Motion skeletons and no video, our pixel model cannot run on it. That number is therefore a reference for what skeleton methods reach with clean sensor skeletons, not a head-to-head result. The fair comparison runs this pipeline on the same RGB benchmarks, with landmarks extracted by the same on-device tracker from On-device Real-time Custom Hand Gesture Recognition. It gets the same hyperparameter-search budget as our model, roughly fifty trials each.
For ChaLearn ConGD, the file reports a top challenge mean Jaccard of 0.6103 in the ChaLearn LAP results paper. We treat it as the published reference to approach.
Three ablations isolate the key ingredients. Removing the local motion code tests whether it carries fine-gesture accuracy. Collapsing the spatial heatmap to a single whole-frame temporal score, which gives an anchor-free temporal detector with no hand localization, tests whether the spatial peak matters. Replacing learned temporal peaks with fixed 32-frame windows tests the anchor-free temporal claim.
Why not just…?
Why not just run MediaPipe landmarks into a graph network? This is the real threat, and we expect it to win at close range with clean tracking. The evidence against it is specific to the hard regime. De Smedt et al. report that skeleton initialization fails when hands close or move quickly. HaGRID reports landmark degradation at 1–4 m and under poor lighting. On-device Real-time Custom Hand Gesture Recognition reports a bias toward static shape. The distance-and-blur stress test below settles it. If the landmark pipeline matches us across the whole range, our accuracy claim fails, and only the efficiency and simplicity argument would remain.
Why not just keep Köpüklü's two-stage gated design? It works, at 91.04% Levenshtein on EgoGesture. Its costs are the redundant re-processing of overlapping windows, a hand-tuned gate, and a fixed 32-frame horizon. It also has no notion of which hand or which person is gesturing. The FLOP-matched comparison and a speed-stratified evaluation, which splits test gestures by duration quartile, will show whether removing windows actually helps on fast and slow gestures. If it does not, the efficiency gain alone must justify the method.
Why not just queue static detections, as HaGRID suggests? Dynamic gestures often share static shapes along their path. In de Smedt et al., adding trajectory and rotation histograms gave a 15% gain over shape features alone. Queues also fire on transient poses during the hand's approach and retraction. The naive baseline measures this directly through false positives per minute of non-gesture motion.
Why not just use an anchor-free temporal detector on whole-frame features, skipping hand localization? This is plausible on single-user egocentric data like EgoGesture, where there is one dominant pair of hands. It is the second ablation above. We expect the spatial peak to matter at long range, where the hand is a few percent of the frame, and when several hands are in view. If the ablation ties our model on all benchmarks and stress tests, we will drop the spatial claim and report the simpler model.
Proposed Experiments
Training uses spatial pretraining on HaGRID, then fine-tuning on EgoGesture, nvGesture and ConGD separately, all in RGB only. Online metrics are Levenshtein accuracy on EgoGesture and nvGesture, mean Jaccard on ConGD, detection rate, false positives per minute on non-gesture segments, early-trigger frames, and on-device latency on a Pixel-class phone. That last metric allows comparison with the 16.76 ms per frame of the MediaPipe pipeline.
Three checks decide the proposal.
The first is an efficiency check. Our model must reach the gated baseline's EgoGesture Levenshtein accuracy within one point at lower measured compute.
The second is a robustness check. We downscale and motion-blur EgoGesture and ConGD test video to simulate apparent distances up to roughly 4 m. We also record a small evaluation set of about 20 participants at labelled distances from 0.5 to 4 m, with two-handed and crossing-hand gestures. On both, we plot accuracy against apparent hand size for our model and the landmark pipeline. The claim holds only if our curve stays flatter and crosses above the landmark curve within the realistic range.
The third is a fine-gesture check. Removing the motion code should cost more on fine finger classes, such as the pinch and grab families in EgoGesture and nvGesture, than on coarse swipes. Removing the code should also increase the static-shape bias, measured as confusion between gestures that share a hand shape but differ in motion.
If all three hold, the result is a single, phone-scale detector that knows which hand issued which command and when, without needing a skeleton.
References
- HaGRID -- HAnd Gesture Recognition Image Dataset
- Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks
- SHREC 2021: Track on Skeleton-based Hand Gesture Recognition in the Wild
- On-device Real-time Custom Hand Gesture Recognition
- Spiking CenterNet: A Distillation-boosted Spiking Neural Network for Object Detection
- CenterMask: Single Shot Instance Segmentation With Point Representation
- MASA: Motion-Aware Masked Autoencoder With Semantic Alignment for Sign Language Recognition
- Toward More Robust and Real-Time Unmanned Aerial Vehicle Detection and Tracking via Cross-Scale Feature Aggregation Based on the Center Keypoint
- Results and Analysis of ChaLearn LAP Multi-modal Isolated and Continuous Gesture Recognition, and Real Versus Fake Expressed Emotions Challenges
- Skeleton-Based Dynamic Hand Gesture Recognition