Searching Small Sign Recognizers That Still Work on New Signers

A sign language recognizer that runs on a phone or a microcontroller has to be small. A recognizer that is useful has to work for people it never saw during training. These two requirements probably conflict, and nobody has measured how. The usual way to shrink a model is to scale down its width or depth, prune it, or search for an architecture under a latency budget. In every case the smaller models are compared on a validation set whose signers also appear in training. A small model has little capacity, so it may fall back on the cheapest cues that work, such as a signer's habitual pace, body build, or resting posture. Those cues hold up on familiar signers and fail on new ones. If so, a size-constrained search scored on the usual validation set will reliably choose the wrong architectures.

We propose to make the architecture search itself signer-aware, in two ways. First, architecture choices are scored on signers held out from weight training. Second, the network is trained to predict sub-lexical "phonological" properties of each sign: handshape, location, movement, finger selection, and sign type, which are the building blocks signs are made of. These properties stay the same from one signer to the next, so they act as auxiliary tasks that keep a small network attending to linguistic content. The search also decides where the shared network should split into property-specific branches, and which operators to use, on measured latency for real edge hardware. The central question is not "can NAS compress a sign model." The question is whether signer-held-out selection and phonological multi-task supervision change which small architectures win, and by how much on unseen signers.

Core Claim

We make three claims, each falsifiable. The first is that the gap between seen-signer and unseen-signer accuracy widens as the hardware budget tightens, so compression costs more on new signers than on familiar ones. The second is that architectures ranked by ordinary validation accuracy and architectures ranked by held-out-signer accuracy disagree substantially under tight budgets, so standard hardware-aware search selects signer-overfit models. The third is that auxiliary phonological heads, with their branch points searched rather than fixed, recover most of the lost unseen-signer accuracy at a fixed on-device latency. If the first two claims fail, the search component is unnecessary, and the proposal reduces to "add phonological heads to a small model." We will report that outcome as it stands.

Relation to Prior Work

The phonological framing comes from two papers that motivate this proposal. WLASL-LEX annotates 800 ASL glosses with six phonological properties. It shows that skeleton-based spatio-temporal graph networks (models that pass messages along the joint graph over time) predict these properties on unseen signers, and even on unseen signs. Its baselines are mostly per-property models, and it names multi-task learning as the natural next step without building it. PHONSSM does build a compositional model. It uses a hand-designed four-subspace factorization with orthogonality losses and a bidirectional state-space temporal model, and reports 72.1% on WLASL2000 at 260 samples/second. Its topology is fixed by hand and it reports no on-device latency. We reuse both papers' inductive bias, but we search over how much of it to keep and where it branches under a hardware budget.

The search machinery comes from hardware-aware NAS. EDNAS is the closest existing work. It searches which layers a multi-task dense-prediction network should share, measures latency on a Coral Edge TPU, and finds that dense fused blocks beat theoretically cheaper depthwise ones. But it works on 2D segmentation and depth, uses i.i.d. splits, and never asks about subject generalization. DUCCIO supplies our constraint handling. It uses a differentiable search with a penalty that switches on only when memory or latency exceeds a target, and reports 54.2% lower latency and 87.4% lower memory on Visual Wake Words. It is single-task and also uses i.i.d. splits. Latency lookup tables from MoGA and the evidence from HAT and the co-design paper constrain the design: FLOPs do not predict device latency, and the best shapes differ across hardware. For that reason we measure latency per target rather than counting operations.

Two papers motivate the evaluation. Rethinking Sign Language Translation shows that a gloss-free translator falls from 21.44 to 10.53 BLEU-4 when signers are properly held out. SDDA shows that appearance augmentation helps unseen signers on RGB video. Its approach is training-time only and orthogonal to architecture. Neither paper studies how this gap behaves as models shrink, which is the open question here.

Research Program

The input is 2D skeletal keypoints. WLASL-LEX found 2D coordinates with confidence scores more reliable than 3D reconstructions. We budget only the recognizer, and report the cost of a separate pose estimator honestly rather than hiding it. The search space mixes operators with different hardware profiles, which addresses an open tension: graph convolutions isolate pose well but map poorly onto edge accelerators. The candidate operators are:

  • graph convolutions over the hand and body skeleton;
  • dense temporal 1D convolutions over flattened joints, which suit accelerators;
  • small attention blocks, as in the Signer-Invariant Conformer;
  • a lightweight state-space layer.

Channel widths and depths are searchable. The search also chooses where each phonological head attaches to the shared trunk, and whether properties share a branch. Fine-grained properties like flexion may need to split off early from hand-only features. Location and movement may share late body-level features.

Let $\alpha$ denote the architecture parameters and $w$ the network weights. Weights are trained on signer set $S_{\text{tr}}$. Architecture parameters are updated on a disjoint signer set $S_{\text{arch}}$ using

$$ \mathcal{L}_{\text{arch}}(\alpha) = \mathcal{L}_{\text{gloss}}^{S_{\text{arch}}} + \sum_{k} \beta_k \mathcal{L}_{k}^{S_{\text{arch}}} + \sum_j \lambda_j \max(0, R_j(\alpha) - T_j). $$

Here $\mathcal{L}_k$ is the loss of phonological property $k$, $\beta_k$ its weight, and $R_j(\alpha)$ the predicted value of hardware metric $j$ (latency or peak activation memory) from a per-device lookup table, with target $T_j$. The last term is DUCCIO's penalty.

Splitting by signer means architecture gradients reward only structure that transfers to new people. This is the key departure from EDNAS and DUCCIO, which tune architecture on data from the same signers. Some properties, such as sign type, learn fast. Others, such as movement, learn slowly. To keep fast properties from dominating, we use RotoGrad-style gradient-magnitude homogenization from RotoGrad to set $\beta_k$ dynamically. We skip its full feature rotation, whose cost grows quadratically with width and is unaffordable here. Every final discrete architecture is compiled and timed on the device, because relaxed super-networks mispredict latency after operator fusion and quantization.

Phonological labels in ASL-LEX are attached per gloss. Any ASL video whose gloss maps to an ASL-LEX entry therefore inherits them. Training uses WLASL-LEX and the mappable portion of ASL Citizen, which has 52 signers, 2,731 signs, and signer-disjoint test splits. The hardware targets are a Coral Edge TPU, a GAP8-class RISC-V microcontroller, and a mobile CPU. Each has budgets at three tightness levels.

Baselines

The naive reference is a skeleton ST-GCN gloss classifier, trained and selected in the usual way, then uniformly width- and depth-scaled and INT8-quantized to each budget. It shows what compression does to unseen-signer accuracy when nobody intervenes. It shares the same keypoints, training schedule, and per-budget tuning budget as every other method.

The strongest existing method is PHONSSM, which already uses phonological factorization and is efficient. We will reimplement it from the paper and scale it to each budget the same way. Its reported 72.1% Top-1 on WLASL2000 comes from a split whose signer overlap is not documented in the file. We found no reported result on Lacuna for PHONSSM on a signer-disjoint ASL Citizen split, nor for any sign recognizer on our target devices. We will therefore produce these numbers ourselves. The external anchor on ASL Citizen is the I3D RGB baseline, at 63.1% Top-1 and 90.8% Recall-at-10, reported by ASL Citizen. We treat it as a reference point, not a budget-matched competitor. Beating PHONSSM means higher unseen-signer Top-1 at equal or lower measured device latency, across three seeds and at least two of three budgets per device.

The strongest search baseline is DUCCIO itself: single-task gloss classification, with architecture parameters tuned on a random validation split that shares signers with training. It gets the same search space, the same number of GPU-hours, and on-device verification. Two ablations isolate our ingredients. One is DUCCIO with signer-split architecture updates but no phonological heads. The other adds phonological heads with branch points fixed at the trunk's end. A third ablation removes gradient-magnitude homogenization.

Why not just…?

Why not just shrink PHONSSM or an ST-GCN and quantize it? This is the most serious threat. If uniformly scaled PHONSSM traces the same unseen-signer Pareto front as our searched models, the search adds nothing. Prior evidence that post-hoc shrinking misses device behavior comes from other domains: EDNAS on Edge TPU and DUCCIO on GAP8 memory. It may not transfer to small skeleton models. The scaled-PHONSSM baseline settles this directly, and we will report a null result if it comes.

Why not just add phonological auxiliary heads to a fixed small model and skip NAS? This could capture most of the gain. The fixed-branch-point ablation, combined with the scaled baselines plus heads, measures exactly how much searching branch points and operators adds beyond the auxiliary supervision. If the remainder is within seed noise, our conclusion becomes "use heads, select on held-out signers," and the search is unneeded.

Why not just run standard hardware-aware NAS and select the final model on held-out signers? Post-hoc selection only reorders candidates that the search already produced while optimizing for seen signers. Our second claim predicts that, under tight budgets, few of those candidates generalize well. The signer-split versus random-split DUCCIO comparison tests this, together with the rank-correlation analysis below.

Why not just augment signer appearance, as SDDA does? SDDA works on RGB pixels, while our input is skeletons. Body-proportion and pace augmentation of keypoints is the skeleton analogue, so we will apply it to every method equally. It is complementary to architecture choice, not a substitute. Since it is applied to all methods alike, it cannot change their relative ordering.

Proposed Experiments

The first decisive check is the gap curve. For the scaled ST-GCN and PHONSSM baselines, we plot seen-signer minus unseen-signer Top-1 against measured latency on each device, on ASL Citizen's signer-disjoint split and on WLASL-LEX's phoneme-level split. Claim one predicts the gap widens as latency falls. If the gap stays flat, the rest of the proposal loses its motivation, and we will say so.

The second check is ranking disagreement. We sample about 200 discrete sub-networks from the trained super-network and compute their accuracy on random-split validation and on held-out signers. We then report the Spearman correlation between the two rankings at each budget. Low correlation under tight budgets would show that standard search optimizes the wrong quantity.

The third check is the Pareto comparison. We compare unseen-signer Top-1 and Recall-at-10 against on-device latency and peak memory for our method, the three baselines, and the ablations. We also report per-property accuracy on WLASL-LEX's gloss-level split, which covers unseen signs and unseen signers, against ST-GCN's reported 83.2% versus 77.3% on Major Location. For transfer, the best architectures found for ASL are retrained on AUTSL, which has signer-disjoint splits but no phonological labels. This tests whether the searched structure helps even when the auxiliary heads are unavailable. We will also record which operators the search picks on each device. Whether graph operators survive on the Edge TPU, or are replaced by dense temporal convolutions, is a direct empirical answer to the open tension between graph-based invariance and accelerator-friendly execution.