Whose "Go"? Annotator-Aware Intent Gating for Compositional Gesture Control of a Soft-Rigid Hand

Gesture interfaces for robots usually recognize only a handful of commands. One known way to grow the vocabulary is to treat gestures like words built from a small alphabet of hand shapes. A recognizer learns ten static shapes and decodes sequences of them, so the vocabulary grows exponentially without collecting data for every sequence. This note argues that the obstacle to scaling these systems is not telling hand shapes apart. It is the decision that comes before that: is the person issuing a command right now, and where does it start and end? That decision is the most subjective label in the pipeline. In practice one person draws it. It sits at the root of every hierarchical recognizer, so any bias it carries repeats at every step of a multi-step command.

We propose to measure this single-annotator bias directly and to replace the root "command / not command" decision with a model that separates two things. The first is shared visual evidence of intent. The second is each annotator's personal threshold for calling something intentional. At deployment, the threshold is calibrated to the end user, who is the only annotator whose judgment matters there. The testbed is a gesture-commanded hybrid soft-rigid robotic hand. We chose it because its compliant shell makes unintended activations safe to observe in closed loop, and its rigid core makes skill execution repeatable. Failures can therefore be attributed to the command layer rather than to the hardware.

Core Claim

Compositional gesture recognizers are hierarchies, whether or not they are described that way. The best example is Talking With Your Hands, which trains 2D CNNs on 10 hand-shape "phonemes" plus preparation, retraction and no-gesture classes and decodes sequences with a modified Viterbi search. Its first branch is "gesture or not". Its second is "which phoneme". It reports 98.47% accuracy on basic classes and 94.69% on 810 three-step gestures on the Scaled Hand Gestures Dataset (SHGD).

If $\epsilon_r$ is the per-step error of the intent/boundary decision and $\epsilon_\ell$ the per-step error of phoneme identification, and steps fail roughly independently, a gesture of $s$ steps is recognized with probability

$$P_{\text{seq}}(s) \approx \left[(1-\epsilon_r)(1-\epsilon_\ell)\right]^{s}.$$

The reported numbers fit this compounding: $0.9847^3 \approx 0.955$, against the reported 0.947. Extending to five-step gestures, the 65,000-command regime the paper extrapolates to, therefore depends on shrinking per-step error.

Our claim is that a large and unmeasured share of $\epsilon_r$ is not model error. It is disagreement about where intent begins, frozen into the training labels by whoever annotated them. Correcting it needs an annotator-aware root, not a bigger phoneme classifier.

The root model is deliberately simple. A shared network produces an intent score $g(x)$ for a window of frames $x$. Annotator $a$ calls it a command with probability

$$p_a(\text{command}\mid x)=\sigma\!\left(\frac{g(x)-\tau_a}{\kappa_a}\right),$$

where $\sigma$ is the logistic function, $\tau_a$ is annotator $a$'s threshold and $\kappa_a$ is how sharply they apply it.

This is a scalar, temporal analogue of the idea in Transformer-based Annotation Bias-aware Medical Image Segmentation. That paper learns per-annotator query vectors and a structured error distribution to separate rater preference from rater noise in optic disc/cup segmentation. We keep its separation of preference from noise but apply it to the binary root of a temporal gesture hierarchy rather than to pixels. At deployment, $g$ is frozen and $\tau_{\text{user}},\kappa_{\text{user}}$ are fit from a few user-confirmed examples. The model defers, holding the hand still and asking for a repeat, when $g(x)$ falls inside the spread of training thresholds.

Research Program

The work has three parts. The first is diagnostic and is worth publishing even if the method fails.

Measuring the bias. We take SHGD's continuous infrared/depth streams and a subset of HaGRID clips that include its "no gesture" class. Seven annotators independently relabel intent onsets, offsets and incidental-motion segments. Seven is enough to simulate 1-, 3-, 5- and 7-annotator consensus following the protocol of Influence of Inter-Annotator Variability on Automatic Mitotic Figure Assessment. That study trained detectors on 12 pathologists individually and in consensus groups and found an interquartile range (IQR) of F1 of 0.038 for single-annotator models, falling to 0.016 at three annotators, with only about +0.013 F1 beyond that.

We ask whether gesture-boundary labels show comparable volatility. We also ask a question specific to hierarchies. If the coarse tree over phonemes is learned from a flat model's confusion matrix, as in HD-CNN, does the tree's topology change when a different single annotator's labels are used? HD-CNN learns its two-level hierarchy by spectral clustering of confusion. It assumes one objective label per image, and our test checks whether that assumption survives subjective temporal labels.

The annotator-aware root. We train $g$ jointly with per-annotator $(\tau_a,\kappa_a)$ on all seven label sets. The phoneme leaf classifiers and the Viterbi decoder are left exactly as in the Köpüklü system, so any change is attributable to the root. Few-shot user calibration chooses which windows to ask about by expected disagreement. This borrows the active-selection idea from Personalizing Gesture Recognition Using Hierarchical Bayesian Neural Networks, which models per-performer weight offsets from a global prior and adapts with 2–5 labels per class on MSRC-12, ChaLearn 2013 and NATOPS. We differ from that work in three ways. We model the labeler's threshold rather than the performer's movement style. We restrict personalization to one scalar decision rather than all weights. We evaluate on continuous compositional streams rather than isolated small vocabularies.

Closed-loop deployment. The decoded phoneme sequences map onto the skill set of the DeltaHand from Tilde. The DeltaHand is a 12-DOF hand whose Delta fingers combine a rigid PLA core with a soft TPU shell. Tilde trains diffusion policies, with DAgger corrections, for grasping, block sliding and lifting, ball rolling, cap twisting, syringe pushing and shape insertion. A two-step command selects skill and object, and a third step confirms or aborts. We use Tilde's policies as fixed executors, retrained on its published recipe if necessary.

Each participant later reviews their session video and marks what they meant. This produces the deployment ground truth: the user's own intent, not any annotator's. Hardware is the application regime here, not the object of study. The soft shell lets us record false triggers mid-task without damage. The rigid core's repeatable kinematics keep the executor's failure rate stable enough to subtract out.

Baselines

The naive reference point is the Köpüklü pipeline trained on one annotator's boundary labels, which is the standard practice. We train seven copies, one per annotator. Following Wilm et al., we report the spread of per-step boundary F1, of $P_{\text{seq}}(s)$ for $s=3$ and $s=5$, and of user-intent false-trigger rate. The cited reference numbers are the reported 98.47% basic-class and 94.69% 810-gesture accuracies on SHGD from Talking With Your Hands. For five-step sequences, no reported result was found on Lacuna. For closed-loop execution, the executor ceiling is Tilde's reported 90% average autonomous success over seven tasks, so command-layer errors must be read relative to that figure.

The strongest competitor is the same pipeline trained on three-annotator majority-vote boundaries. Wilm et al. show that this alone captures most of the stabilization from consensus, and HaGRID institutionalizes it with hard/soft voting. Its ResNet-18 reaches 97.5% F1 on 18 static classes, but that number concerns shape classification, not intent boundaries. For fairness, the consensus model, the annotator-aware model and all single-annotator models share the backbone (MobileNetV2, as in Köpüklü), the Jester pretraining, the leaf classifiers, the decoder, the annotation hours, and an identical hyperparameter search budget. Every model is also given the same few user-confirmed examples at deployment. The consensus model uses them to re-tune a global decision threshold. Beating this baseline means a lower user-intent false-trigger rate and a lower missed-command rate at matched deferral rate, measured on held-out users across sessions, with the gap exceeding the across-user IQR.

The key ablation removes the per-annotator thresholds and trains $g$ on soft (averaged) labels. The test model then has the same information as ours but no explicit threshold to recalibrate. A second ablation keeps the annotator-aware root but trains the coarse phoneme tree from single-annotator confusion, to isolate whether tree instability matters downstream.

Why not just…?

Why not just use three annotators and majority vote? This is a real threat. Wilm et al. found that diminishing returns set in after three raters, and that consensus-building mattered more than which raters were chosen: 0.756 F1 for the three least agreeable against 0.751 for the three most agreeable. Consensus removes random annotator error. It cannot represent the fact that the end user has their own threshold, and our deployment metric is defined by that user. If majority vote plus a re-tuned global threshold matches our model on user-intent error, consensus is sufficient, and we will report it as the practical recommendation. The diagnostic study stands either way.

Why not just add temporal smoothing or hysteresis to the single-annotator model? Köpüklü's Viterbi decoding already smooths. Smoothing removes jitter around a boundary. It does not move a boundary that is systematically early or late, nor relabel a habitual incidental motion as non-intentional. The annotator-swap experiment settles this: train on annotator A, score against annotator B with and without extra smoothing. A persistent offset that smoothing does not close means the error is threshold bias, not noise.

Why not drop the hierarchy and use one flat classifier with a no-gesture class and a cost-sensitive decision rule? With a calibrated posterior and a known cost matrix, a flat Bayes-risk rule is optimal for isolated decisions, so the hierarchy must earn its place. It earns it through composition. The leaves are reused across exponentially many sequences, and only the root needs user calibration. A flat model over 810 or 65,000 sequences cannot be trained without sequence data, which is the bottleneck Köpüklü removed. The constraint we take seriously comes from Automatic classification of eating conditions from speech. Its Food/No-Food-then-subtype SVM hierarchy rose from 60.9% to 73.7% unweighted average recall (UAR) in training cross-validation but gained only 1.4% UAR on test without feature selection. Hierarchy gains can therefore vanish out of sample. We include a flat phoneme-plus-no-gesture classifier with Bayes-risk decoding at matched compute, evaluated on single-step commands, to measure this directly.

Proposed Experiments

The first decisive check is whether the bias exists at a scale that matters. If single-annotator boundary models on SHGD show IQR in $P_{\text{seq}}(5)$ well above the across-seed IQR of one annotator's models, then annotator choice, not training noise, is shaping long-command accuracy. If the HD-CNN-style confusion trees also change topology across annotators, the problem extends beyond the root. If the spread is no larger than seed noise, the premise fails, and we stop at the diagnostic paper.

The second check is decomposition. Using the compounding relation above, we estimate $\epsilon_r$ and $\epsilon_\ell$ separately from step-level errors. We test whether the annotator-aware root lowers $\epsilon_r$ against consensus on held-out annotators' labels, and whether the gain grows from $s=3$ to $s=5$, as compounding predicts.

The third check is closed-loop. Twelve new participants, none of them annotators, each run two sessions of 30 multi-step commands on the DeltaHand under each condition in counterbalanced order: single-annotator, consensus, soft-label ablation, and annotator-aware. We record user-intent false triggers per minute, missed commands, deferral rate, and end-to-end task success normalized by Tilde's 90% executor ceiling. Cross-session evaluation also probes a limitation the file flags for both TAB and the hierarchical Bayesian model: they assume rater style is static. If $\tau_{\text{user}}$ fitted in session one fails in session two, that drift is a finding in its own right, and it motivates online re-estimation of a single scalar rather than of a whole network.