Hiding Commands in Fidgets: Telling Intentional from Spontaneous Micro-Gestures

People constantly make small, unremarkable movements. They touch their collar, rub a finger along the jaw, or tap the table. Bystanders ignore these motions because they look like ordinary fidgeting. That makes them an attractive input vocabulary for glasses, headsets and ambient sensing in public. Human-computer interaction research keeps finding that users reject visible gestures even when those gestures are physically easy, and that people prefer inputs that disappear into normal behavior. The catch is that if a command looks exactly like a fidget, the recognizer will fire every time the user fidgets. Our proposal is to turn this tension into a measurable problem. We collect paired recordings of the same micro-gestures performed spontaneously and on purpose. We ask third-party observers whether they can tell the two apart. We then train a lightweight model over joint positions across time (a spatio-temporal model) to make the distinction from timing cues, such as onset sharpness, velocity profile and hold duration, that observers may not perceive.

The central question is whether a gap exists between what humans see and what a model can learn. If some fidget-like gestures are indistinguishable to bystanders but separable by a model at a false-trigger rate below one per hour, then social acceptability and reliable recognition stop being opposed. If no such gestures exist, that is a useful negative result about where the limits of camouflaged input lie.

Core Claim

Current gesture recognizers and current acceptability research pull in opposite directions. The closest technical work, Enabling Hand Gesture Customization on Wrist-Worn Devices, runs a "sanity check" that rejects user-defined gestures that resemble daily movements. It suppresses false positives by pushing the vocabulary away from natural behavior, and reports 0.6 false positives per hour. We propose the opposite move: deliberately choose gestures that resemble natural behavior, because that resemblance is what makes them socially invisible, and recover reliability through intent-sensitive temporal features.

The claim is not that a new backbone wins a benchmark. It has three parts:

  1. The relevant acceptability measure for this design is observer discriminability, meaning whether a bystander can tell a command from a fidget, rather than the performer's self-report.
  2. The relevant recognition measure is intent discrimination within the same gesture category, rather than discrimination between categories.
  3. A per-gesture tradeoff between these two quantities can be estimated and used to pick a vocabulary.

Research Program

The work has three components: a measurement of natural base rates, a paired dataset with an observer study, and an intent-discriminating model.

Natural base rates from existing data. The two large micro-gesture benchmarks treat these movements purely as involuntary leakage of emotion or stress. iMiGUE annotates 32 categories from tennis press conferences. SMG records 16 to 17 categories with RGB, depth and skeletons during stressed and unstressed storytelling. Neither benchmark asks whether these movements could serve as deliberate input; that question motivates this proposal. Their annotations do, however, supply the first quantity we need: how often each gesture occurs naturally per hour of conversation, which we write as $\lambda_g$ for gesture $g$. A recognizer that simply detects category $g$, with no notion of intent, will trigger at roughly $\lambda_g$ spurious events per hour. This is the "do nothing" reference, and it can be computed before any new data is collected.

Paired intentional and spontaneous dataset. We would record about 30 participants with a front-facing RGB-D camera that matches the SMG capture setup, so that pretraining transfers. The study has two blocks:

  1. Spontaneous block. Participants engage in about 20 minutes of conversation and storytelling, following SMG's elicitation protocol, which provides natural instances of each gesture.
  2. Command block. Participants continue conversing, but a wrist vibration periodically cues them to issue a command using a designated fidget gesture.

The command block has two conditions. In the "free" condition, the participant performs the gesture however they like. In the "signature" condition, the participant performs it with a small temporal key, such as a 600 ms hold or a double touch. The signature condition tests whether a timing convention can raise how well the model separates the two classes without raising how well observers can tell them apart.

Candidate gestures are drawn from the categories in iMiGUE and SMG, plus one deliberately "non-natural" tiny pinch. The pinch is the vocabulary the sanity-check approach would produce, and it serves as the control.

Observer study. Independent raters view short clips and judge whether each clip shows a command. Following the observer-perspective gap identified in the acceptability literature, we score each gesture by how far observers fall above chance. We use the sensitivity index $d'_{\text{obs}}(g)$, a signal-detection measure where zero means observers are at chance. Performers also rate their own comfort, which lets us compare the observer view with the performer view.

Intent-discriminating model. The backbone is the Spatio-Temporal Multilayer Perceptron. It alternates layers that mix information across body joints within a frame with layers that mix information across frames, and it runs in 0.66 ms on a 2080Ti. We reuse it as a cheap method that keeps temporal order. It has only been tested on large, whole-body gestures, so applying it to micro-motions is itself a test.

We would extend the skeleton with facial landmarks following Towards Fine-Grained Emotion Understanding via Skeleton-Based Micro-Gesture Recognition. That work expanded the skeleton to 41 joints and argued that graph models with a fixed joint connectivity treat low-amplitude motion as noise.

The key training ingredient is an intent-contrastive loss. Each intentional instance is paired with spontaneous instances of the same category and the same participant as hard negatives. This forces the model to rely on dynamics rather than on which joints are involved. We would pretrain on iMiGUE and SMG category labels, then fine-tune on the paired data. For deployment, we report the expected false-trigger rate per hour as

$$F(g) = \lambda_g \cdot \mathrm{FPR}_g ,$$

where $\mathrm{FPR}_g$ is the model's false-positive rate on spontaneous instances of $g$ at a fixed recall on intentional ones. Gestures are then plotted by $(d'_{\text{obs}}(g), F(g))$, and the chosen vocabulary is the set of gestures that are best on both axes at once.

We also report FLOPs and latency against the envelope set by Helios: 164 MFLOPs and under 350 mW on an eyewear processor. On-device deployment itself is out of scope.

Baselines

Do-nothing reference. The do-nothing reference is a category recognizer with no intent modeling. Its false-trigger rate equals $\lambda_g$ measured from SMG and iMiGUE annotations. Any proposed system must sit well below this line at matched recall to be worth using.

Strongest category classifier. This baseline is Prototype Learning for Micro-gesture Classification. It is a PoseConv3D model, a 3D convolutional network over stacked joint heatmaps, with a pathway for RGB frames, a pathway for skeleton heatmaps, and cross-attention between them. It also includes a module that refines predictions toward per-class prototypes. It reports 70.25% Top-1 on iMiGUE as an ensemble and 67.91% as a single model. It is the strongest competitor because it is the most discriminative micro-gesture model available. If intent can be read from appearance plus pose, this model should read it.

We would fine-tune the single-model version on our paired data, with intent added as a label and spontaneous same-category clips as negatives. It receives the same pretraining data and the same hyperparameter search budget as our model. Our model beats it if it matches or exceeds its intent discrimination, measured by the area under the ROC curve on held-out participants, at a fraction of the compute.

Strongest online spotter. For continuous detection, the reference numbers on SMG are the 27.57 F1 of the top 2024 MiGA challenge entry and the 14.34 F1 of Micro-gesture Online Recognition using Learnable Query Points. The latter combines Mamba, a linear-time sequence model, with self-attention over learnable temporal query points. We would rerun the Learnable Query Points system with intent labels on our continuous recordings. We report events per hour rather than challenge F1, because false triggers per hour is the quantity users actually experience. There is no reported result on Lacuna for intentional-versus-spontaneous discrimination on any benchmark, so these baselines are established inside this study.

Ablations. Two ablations isolate the key ingredient:

  1. Without intent-contrastive negatives. Intent is learned as an ordinary binary label, with negatives drawn from other categories rather than from the same category and participant. This tests whether the paired construction matters.
  2. With temporal order destroyed. Input is collapsed into a static motion image, analogous to Helios's time surfaces, which compress away the temporal order Helios needed to tell a single pinch from a double pinch. This tests whether intent really lives in the dynamics.

Why not just…?

Why not just follow the sanity-check approach and pick gestures unlike daily movement? That is what Enabling Hand Gesture Customization on Wrist-Worn Devices does. It works for false positives, but it may push the vocabulary toward conspicuous motion, consistent with Understanding the Design Space of Mouth Microgestures, which found that users rejected visibly noticeable actions. This is a real threat if tiny non-natural gestures such as a 1 cm pinch are already invisible to observers. In that case camouflage buys nothing. The pinch control condition settles this directly: if its $d'_{\text{obs}}$ is as low as that of the fidget gestures and its $F$ is lower, the sanity-check approach wins and we would say so.

Why not just add an activation step, such as gaze dwell or a wake gesture, before each command? An activation step converts intent detection into a two-step sequence. Make Interaction Situated found gaze good for browsing but in need of a physical confirmation for action, which suggests gaze alone is not a reliable activation step. Still, a combination of gaze and fidget could beat a fidget alone. We include it as an arm and measure both its false triggers and its added latency and observer discriminability. If it dominates, then the temporal-signature idea is unnecessary.

Why not just fine-tune the best existing classifier with spontaneous clips as a negative class? This is the Prototype Learning baseline above, and the ablation without intent-contrastive negatives approximates it with our backbone. If it matches the full model, the contribution reduces to the dataset and the observer measurement. That outcome is still publishable, but it would change the claim.

Why not just use invisible sensing such as wrist muscle sensors? BeyondVision reaches 95.66% on 1 cm pinches with 16-channel forearm EMG (electromyography, which measures muscle electrical activity). This requires visible wristband hardware and per-user calibration, which the acceptability literature associates with stigma. Our question applies to camera-based ambient and headset sensing. The intent-from-dynamics question also transfers directly to muscle signals in future work.

Proposed Experiments and Decisive Checks

First experiment, before new data. Compute $\lambda_g$ for every category on SMG continuous streams and iMiGUE videos. This identifies which fidgets occur rarely enough to be plausible command candidates.

Reproduction. Reproduce MS-G3D's 64.75% on SMG classification and Prototype Learning's single-model iMiGUE result, to confirm that our pipeline is sound.

Paired-data analyses. On the new data, with all splits separated by participant, we report:

  1. Intent discrimination (AUC) per gesture for each model.
  2. $F(g)$ at 80% recall.
  3. Observer $d'_{\text{obs}}(g)$.
  4. Performer comfort ratings.

Decisive checks. The premise is supported if at least one gesture with $d'_{\text{obs}}$ statistically indistinguishable from zero reaches $F$ under one event per hour at 80% recall. One per hour is the order of the 0.6 per hour that Xu et al. considered acceptable.

The premise is refuted, and camouflaged input reported as infeasible in this setting, if either of two things happens:

  1. Every gesture separable by the model is also separable by observers. This would mean the model has no perceptual advantage to exploit.
  2. The model fails to beat $\lambda_g$ by an order of magnitude.

Diagnostic checks. The signature condition shows whether a timing key moves gestures toward the desirable region of low observer discriminability and low false triggers. The ablation with temporal order destroyed tells us whether that improvement comes from temporal structure, which is the part of the result that ties social acceptability to spatio-temporal feature extraction.