A camera that can move, such as one on a drone, a robot arm, or a switchable multi-camera rig, should point where it learns the most about what is being signed. "Learning the most" should not mean "reconstructing the whole hand most accurately." Sign languages build words from a small set of reusable parts: handshape, location on the body, movement path, and palm orientation. At any moment during a sign, most of these parts are already settled by what the camera has seen. Often only one remains ambiguous, for example whether the thumb is tucked or extended, or whether the palm faces in or out, and only a few viewpoints can resolve it. We propose an active viewpoint selector that scores candidate cameras by how much each would reduce uncertainty about the sign's identity. It works through a recognizer that factorizes signs into these components and through the lexicon, which records which component values actually distinguish the remaining candidate words. To choose a view before the hand arrives there, the selector forecasts articulated hand motion with a discrete library of learned motion tokens rather than a linear extrapolation.
The opportunity follows from two findings that have not been connected. Monocular sign recognizers struggle most where depth and self-occlusion matter. Movement is the hardest phonological property to predict in WLASL-LEX, and monocular 3D lifting there underperforms 2D keypoints. Separately, active human motion capture shows that choosing views orthogonal to the motion plane cuts error substantially (ActiveMoCap). The research file found no work that steers a camera to resolve sub-lexical ambiguity. Every active-vision objective it covers targets whole-scene fidelity or whole-skeleton error.
Core Claim
We claim that a lexicon-aware, component-factorized information-gain objective picks better views for recognition than an objective that minimizes generic pose uncertainty. The advantage should be largest on minimal pairs (signs differing in one component) and under tight view budgets. Concretely, let $S$ be the sign identity, $o_{1:t}$ the observations so far, and $v$ a candidate camera. The selector chooses
$$v^* = \arg\max_v \; H(S \mid o_{1:t}) - \mathbb{E}_{\hat{c} \sim q(\cdot \mid v, \hat{x}_{t+\Delta})}\big[H(S \mid o_{1:t}, \hat{c})\big],$$
where $H$ is entropy and $\hat{c}$ is the vector of component labels the recognizer would output from view $v$. The term $q$ is an offline-learned, view-conditioned confusion model for each component: the probability of each predicted handshape, orientation, location, or movement label given the true label, the camera's angle relative to the forecast hand pose $\hat{x}_{t+\Delta}$, and distance. The posterior over signs factorizes as $p(S \mid o) \propto p(S)\prod_k p(c_k(S) \mid o)$, where $c_k(S)$ is the lexicon's value of component $k$ for sign $S$.
Two consequences make this different from generic uncertainty. First, components already pinned down contribute nothing, so the selector ignores views that would only sharpen a settled handshape. Second, the expectation needs no rendering and no gradients through a renderer, so scoring hundreds of candidate views is a table lookup plus a small sum. This avoids the cost problems flagged for renderer-coupled planners such as GO-PRE and ActiveNeRF.
Relation to Prior Work
The method draws on three lines of work, each in a different role.
The recognizer is the method we reuse. PHONSSM projects skeletal hand features into four orthogonal phonological subspaces and classifies signs hierarchically, reaching 72.1% top-1 on WLASL2000. We adopt its factorization but run it on observations from a chosen view rather than a fixed one. Its paper also constrains us: factorized models can lose to end-to-end models on dense minimal pairs. Our claim is that this loss partly reflects bad viewpoints, which is testable. WLASL-LEX supplies the six-property lexicon labels for 800 signs, which define the prior $p(S)$ and the component structure $c_k(S)$.
The motion forecaster is the second reused method. A Data-Driven Representation for Sign Language Production quantizes continuous 3D sign pose into a codebook of motion tokens: 2,500 tokens cover roughly 4,500 signs in mDGS. That paper uses the tokens for generation from text. We instead train a next-token predictor conditioned on the partial sign posterior, so the forecast hand pose depends on which words are still possible. This replaces ActiveMoCap's linear forecast, which that line of work identifies as failing under abrupt articulatory changes.
The active-vision literature motivates and bounds the design. ActiveMoCap scores views by the curvature of a whole-body pose energy. It is our closest competitor, and we keep its teleportation evaluation protocol. Scene-level planners, including Creat3r with its static triangulated scaffolds and GO-PRE with its rendering entropy over static Gaussians, assume rigid scenes. The non-rigid reconstruction work (4DTAM, Superpoint Gaussian Splatting) shows that dense deforming-surface models are slow (about 1.5 FPS for 4DTAM) and hard to initialize. We therefore deliberately keep non-rigid modeling at the level of articulated hand pose and token forecasts, and do not attempt dense surfaces. Recognition does not need dense geometry, and dense geometry would make planning intractable.
Research Program
The work has four components.
The first is a multi-view signing simulator with derived component labels. The file lists no multi-view sign dataset, so we follow ActiveMoCap's approach of animating motion capture in simulation. We retarget 3D sign pose sequences (the mDGS and PHOENIX14T pose data used by the codebook paper) onto a parametric hand-and-upper-body mesh and render them from a sphere of candidate cameras. Component labels come geometrically from the 3D ground truth: handshape from clusters of finger flexion and selected fingers, orientation from the palm normal, location from wrist position relative to body landmarks, and movement from the token sequence. Deriving labels this way also tests a less rigid, learned phonological schema, which PHONSSM names as an open problem.
The second is the set of view-conditioned confusion models. For each component, we train the factorized recognizer on renders. We then estimate $q$ by binning the camera angle relative to the palm normal and the forearm axis. A sanity check is whether $q$ reproduces known failure patterns, for example orientation confusions concentrating in edge-on palm views.
The third is the forecaster, a small autoregressive model over motion tokens conditioned on the current sign posterior. It predicts hand pose 200 to 500 ms ahead, matching realistic camera repositioning latency.
The fourth is the selector loop. It runs in two settings: teleportation, where any camera in a discrete set can be chosen per time window, and a motion-constrained setting, where a simulated drone has speed limits so that forecasting matters. We then port the confusion models and selector to real multi-view footage in InterHand2.6M. That dataset has 80 to 140 cameras and triangulated 3D joints at 2.78 mm annotation error. We define handshape and orientation classes from its 3D joints and pose a real-data minimal-pair handshape task under per-frame camera choice.
Baselines
The do-nothing reference is a single fixed frontal camera, the standard signer-facing view that WLASL-style data assumes. It runs with the same recognizer and the same training data. A random-view policy and ActiveMoCap's constant-angle following policy share the same view budget and set the naive active floor. The reported floor in the file, on a different task, is random selection at 0.28 m mean joint error versus ActiveMoCap's 0.21 m and an oracle's 0.17 m on MPI-INF-3DHP teleportation. These numbers show the scale of gains active selection can bring, but they are joint error, not recognition accuracy.
The strongest existing method is ActiveMoCap's criterion: choose the view that minimizes predicted 3D joint uncertainty. We reimplement it over the same hand-and-arm skeleton, the same candidate views, the same forecaster input (run once with its linear forecast and once with our token forecast), and the same tuning budget for any weighting hyperparameters. No reported result was found on Lacuna for active view selection measured by sign or phonological-component recognition accuracy, so this comparison establishes the first such number.
Beating it means higher sign top-1 accuracy on the minimal-pair subset at equal view budget, with confidence intervals over signers that exclude zero. Matching it on overall accuracy while beating it on minimal pairs would still count. Matching it everywhere would mean our ingredient is unnecessary.
An upper reference fuses all cameras at every frame, which approximates a passive dense rig, alongside an oracle that picks the single best view in hindsight.
The ablations isolate the key ingredients. The first drops the lexicon and maximizes the summed component entropy, which tests whether knowing which components still separate candidate words matters. The second swaps the token forecaster for linear extrapolation. The third substitutes a monolithic recognizer whose uncertainty comes from dropout sampling, which tests whether factorization is what makes cheap targeted scoring possible.
Why not just…?
Why not just use a fixed multi-camera rig? A dense rig is the right tool in a studio, and InterHand2.6M shows it gives millimeter-level accuracy. It is not available in homes, on robots, or in egocentric settings like Nymeria. Our all-view fusion reference measures exactly how much accuracy a one-camera active policy gives up. If the gap is small at budgets of one or two views, the claim holds. If it is large, the rig wins and we will say so.
Why not just point the camera at the palm? A palm-facing heuristic is a real threat for orientation and handshape. We include it as a baseline, steered by the forecast palm normal. It has no answer when the ambiguity concerns location or movement, where views orthogonal to the motion plane matter, as ActiveMoCap found. It also cannot trade off components. The per-component breakdown will show where the heuristic matches the full objective and where it does not.
Why not just minimize generic pose uncertainty, as ActiveMoCap does? This is the most plausible alternative. On signs with gross differences it may do as well as our method, because good joint estimates imply good component estimates. Our bet is narrower: on minimal pairs, the decisive degrees of freedom are a small fraction of joint error, and a joint-error objective spends views on settled parts of the hand. The minimal-pair split and the no-lexicon ablation settle this directly.
Why not just train a view-invariant recognizer with heavy viewpoint augmentation? Augmentation helps with views the recognizer is given, but it cannot recover information that is occluded in the chosen view. Our confusion models quantify this, because some bins remain confused even after augmented training. We train every recognizer, including the fixed-view baseline, with the same augmentation, so any active gain is beyond what augmentation alone provides.
Proposed Experiments
The primary experiment runs in simulated teleportation with 14 to 26 candidate cameras, mirroring the 14-camera MPI-INF-3DHP protocol. We report sign top-1 accuracy, per-component accuracy, and minimal-pair accuracy at budgets of one, two, and four views per sign, plus the number of views needed to reach 90% posterior confidence.
The decisive checks are three. First, our selector must beat the ActiveMoCap criterion on minimal pairs at budgets of one and two views. Second, the lexicon ablation must lose part of that gain; otherwise the lexicon-awareness claim is wrong. Third, in the motion-constrained drone setting, the token forecaster must beat linear extrapolation on fast movement-class signs, the property WLASL-LEX found hardest.
Transfer to real footage is tested on InterHand2.6M minimal-pair handshape discrimination with per-frame camera choice. We report MPJPE alongside to check that recognition gains do not come from degraded geometry, against InterNet's reported 12.16 mm single-hand MPJPE. Finally, we measure selection latency against GO-PRE's roughly 120 ms per view and Creat3r's roughly 10 s per iteration, expecting lookup-based scoring to be well under both.
References
- WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language
- ActiveMoCap: Optimized Viewpoint Selection for Active Human Motion Capture
- GO-PRE: Goal-Oriented Next-Best-View Selection via Predictive Rendering Entropy for Active 3D Reconstruction
- ActiveNeRF: Learning Where to See with Uncertainty Estimation
- State Space Models are Effective Sign Language Learners: Exploiting Phonological Compositionality for Vocabulary-Scale Recognition
- A Data-Driven Representation for Sign Language Production
- Creat3r: Confidence Reaggregation for Exploration-aware Active 3D Reconstruction
- 4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians
- Superpoint Gaussian Splatting for Real-Time High-Fidelity Dynamic Scene Reconstruction
- InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image
- Nymeria: A Massive Collection of Egocentric Multi-modal Human Motion in the Wild