When people type in virtual reality, a headset hides their hands. The fix that has worked best is also the simplest: draw a small translucent sphere at each fingertip and nothing else. With spheres, error rates come close to those with live video of the hands, and are far lower than with a full 3D hand model or no hands at all. This result, however, was measured with a laboratory motion-capture rig that tracks fingertips almost perfectly. A consumer headset has to find fingertips with its own cameras, which suffer occlusion and motion blur, and training such a tracker for typing has so far required real recordings of hundreds of people wearing motion-capture markers. Synthetic data, meaning rendered images with free and exact labels, is the obvious way to cut that cost. Models trained on renders, though, lose accuracy on real images, a problem called the sim-to-real gap, and no existing synthetic hand pipeline renders fingers striking a surface at typing speed.
This note proposes to close the gap against a yardstick that comes from the interface itself. First, we measure how much tracking error, and which kinds, the fingertip-sphere display can absorb before its typing benefit disappears. Second, we build a synthetic generator of egocentric typing hands. Its tracker is adapted to real images using only touch events from a cheap touchscreen instead of motion capture. The claim is that the error budget is lopsided: users should tolerate slow drift but not flicker, dropouts, or height errors near contact. A synthetic pipeline aimed at the costly error types, and anchored by touch events at the moments of contact, can meet that budget without a motion-capture rig. Mean joint error does not reveal whether it does.
Core Claim and What Is New
Prior work motivates this from two sides that have not been joined. Effects of Hand Representations for Typing in Virtual Reality compared four hand renderings on a physical keyboard using OptiTrack ground truth. It reports a character error rate of 5.1% for video hands, 6.3% for fingertip spheres, 11.5% for an inverse-kinematics hand model and 15.2% with no hands. It never asks what happens when the tracked positions are wrong. Grubert and colleagues' work on text entry with standard keyboards in VR and the virtual office reaches similar conclusions under the same idealized tracking. On the vision side, TouchInsight shows that sensing error, not motor error, is the bottleneck for egocentric surface typing. It reaches 6.3 mm mean spatial error, a touch-timing F1 of 0.99 and 37.0 words per minute (WPM) at a 2.9% uncorrected error rate. It does this by training on 385 participants recorded with OptiTrack and a Sensel Morph pressure pad, and it uses no synthetic data. Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation and RenderIH supply the synthesis methods we reuse: anatomical hand models with forearms, frequency-statistics augmentation, and interpenetration handling. Both evaluate isolated or interacting hands, not hands on surfaces. RenderIH names hand–surface contact explicitly as future work.
What is new is the combination of three pieces. The first is a measured error budget for the minimalist display. The second is typing-specific synthesis: surface contact, fingertip-pad compression, and motion blur at strike speed, rendered from headset-like monochrome cameras. The third is adaptation to real data using touch events as the only real labels. We also resolve a tension the literature leaves open. The vision paper finds that removing the forearm from training images ("floating hands") causes errors that cascade down to the fingertips, while the HCI work finds that users should see only the fingertips. We decouple the two: the tracker is trained and run with full arm context, and the display shows only spheres.
Research Program
The error-budget study. Participants type in VR on a physical keyboard and on a flat table with a projected layout. We follow the mobile email phrase-set protocol of Effects of Hand Representations. Their fingertips are tracked by motion capture, which gives near-perfect positions. Before the spheres are drawn, we deliberately corrupt those positions with controlled errors. There are five error types at graded magnitudes: constant bias parallel to the surface, bias in height above the surface, random jitter from frame to frame, extra latency, and short dropouts concentrated in the 100 ms before a keystroke, where egocentric occlusion is worst. For each type, the output is a curve of character error rate against error magnitude. The budget for that error type is the magnitude at which the sphere condition's advantage over no hands, measured as the gap in character error rate, halves. This study stands on its own. It tells a headset maker what tracking quality buys, and it becomes the target for the vision work.
The synthetic generator. We extend the NIMBLE-plus-forearm pipeline of the domain-gap analysis, keeping its amplitude-spectrum augmentation, which matches the image-frequency statistics of renders to those of real photos. We add what typing needs. Motions are generated from real keystroke logs and a keyboard layout: inverse kinematics places each striking fingertip on its key, and a downstroke velocity profile sets how fast it arrives. A small real motion-capture typing library is used only to check that the generated motions are realistic. Fingertips compress against a rigid plane through a simple soft-pad deformation, and RenderIH-style signed-distance constraints stop fingers from passing through each other and through the surface. Images are rendered as monochrome frames from headset camera positions. Motion blur comes from averaging subframes over the exposure window, and we add rolling-shutter skew and sensor noise. Every frame carries exact labels for fingertip position, height above the surface, contact state, and which finger is touching.
Touch-anchored adaptation. On real data we record only headset video plus touches from an ordinary tablet touchscreen lying on the table. Each touch $k$ gives a time $t_k$ and a surface location $u_k$, but the touchscreen does not report which finger made it. We therefore fine-tune the synthetically pretrained tracker with a loss that assigns each touch to the best-matching predicted fingertip:
$$L_{\text{touch}} = \sum_k \min_{f} \Big( \lVert \Pi(\hat p_f(t_k)) - u_k \rVert^2 + \lambda\, \hat h_f(t_k)^2 \Big)$$
Here $\hat p_f(t)$ is the predicted 3D position of finger $f$ at time $t$, $\Pi$ projects a point onto the surface plane, $\hat h_f(t)$ is the predicted height of that finger above the surface, and $\lambda$ weights the height term. The height term encodes that the touching finger must be at zero height. We also add a mild temporal smoothness penalty on non-contact frames. These labels exist exactly at the moments the budget study is expected to flag as most costly, just before and at contact. The tracker outputs a fingertip position with an uncertainty, trained with the β-NLL loss of TouchInsight (a negative log-likelihood whose weighting keeps the model from inflating its predicted variance to avoid fitting hard samples). This lets the tracker plug into that paper's Bayesian decoder, which combines the tracker's uncertainty, a model of human aiming error, and a language model to pick the most likely text.
Baselines
The naive reference point is the headset's stock hand tracker with no typing-specific training, driving the spheres directly, alongside the no-hands condition. The no-hands character error rate of 15.2% in Effects of Hand Representations is the ceiling any tracker must stay well below.
The strongest existing method is TouchInsight-style training on real motion-capture data, which reports 6.3 mm spatial error, touch F1 of 0.99 and 37.0 WPM at 2.9% uncorrected error (TouchInsight). Its 385-participant dataset is not available to us, so we reproduce the recipe on our own motion-capture recordings. We run a sweep over the number of participants with the same architecture, the same compute, and the same tuning budget as our method. Evaluation is on a held-out test set of participants, both with motion-capture ground truth.
The strongest synthetic competitor is pretraining on generic synthetic hands in the style of SynFrei or RenderIH, followed by the same touch-anchored fine-tuning. On their own benchmarks these report 1.02 cm versus 0.99 cm mesh error on FreiHAND, and a drop from 6.76 mm to 5.79 mm on InterHand2.6M, respectively.
Three ablations isolate our ingredients: the typing generator with no real fine-tuning; removing blur, contact compression, and surface constraints one at a time; and the typing generator with forearms removed, to test whether the floating-hand cascade shows up at the fingertips during typing.
Beating the real-data baseline means one of two things. Either we meet the measured budget for every error type with no motion capture in training, or we match the real-data baseline's contact-window fingertip error while it needs several times more participants. Matching its mean joint error alone does not count.
Why not just…?
Why not just collect more real motion-capture data, as TouchInsight did? This is a real threat, because it plainly works. The argument against it is cost and coverage: rigs, markers, and pressure pads for hundreds of people, in one lab setting. RenderIH's finding that synthetic pretraining matches full-data accuracy with about 10% of the real labels suggests synthesis can substitute for much of it, but that was shown on interacting hands, not typing. The participant-count sweep settles the question. If real-only training with modest numbers of participants already meets the budget, the synthetic generator is not worth its complexity, and we will report that.
Why not just take the stock tracker and snap the spheres onto the surface with smoothing? This may be the most dangerous alternative. If the budget study shows that height error and jitter dominate, then a plane prior plus a temporal filter might fix most of the problem for free. We therefore run exactly this post-processing on every tracker. It cannot recover a fingertip hidden behind another finger, or tell which finger struck, which is where TouchInsight locates the main sensing errors. The contact-window dropout budget is the test that separates the two.
Why not just show passthrough video of the hands? Video hands gave the lowest error, 5.1% versus 6.3% for spheres, in Effects of Hand Representations. Mixed-reality headsets can do this without tracking at all. We include passthrough as a reference condition. Our target is fully virtual scenes, where video breaks immersion and the fingertip positions are needed anyway to detect touches on uninstrumented surfaces.
Why not just use generic domain randomization or frequency augmentation without typing-specific rendering? The generic-synthetic baseline tests this directly. If it matches the typing generator after touch-anchored fine-tuning, then the fine-tuning is doing the work and the rendering additions are unnecessary.
Proposed Experiments and Decisive Checks
The real evaluation set covers roughly twenty held-out participants typing on a keyboard and on a bare table, with motion capture and a touchscreen for ground truth. We report fingertip error split into contact-window and hover segments, error along the surface and in height, touch-timing and finger-identity F1, and, closed-loop, WPM and error rate with the sphere display and the TouchInsight decoder. FreiHAND (dataset) serves as a check that general-pose accuracy does not collapse.
Three results decide the question. First, the budget curves must actually be lopsided. If all error types cost the same per millimetre, the premise fails and mean error is an adequate target. Second, the touch-anchored synthetic model must land inside the budget on real headset video without motion capture in training. Third, in the closed-loop study, our tracker's sphere condition must stay closer to the motion-capture sphere condition than to no hands. That would show that the 6.3% versus 15.2% advantage from Effects of Hand Representations holds up under real egocentric tracking.
References
- Effects of Hand Representations for Typing in Virtual Reality
- Text Entry in Immersive Head-Mounted Display-Based Virtual Reality Using Standard Keyboards
- The Office of the Future: Virtual, Portable, and Global
- TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision
- Analyzing the Synthetic-to-Real Domain Gap in 3D Hand Pose Estimation
- RenderIH: A Large-scale Synthetic Dataset for 3D Interacting Hand Pose Estimation
- FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images