Most systems that recover 3D hand pose from video work in two stages. A 2D detector finds joint locations in each frame, and a separate "lifting" network turns those 2D points into 3D. The lifter never sees the image. When a hand grips a mug or two hands interlace, some fingers are hidden. The detector then guesses, often with high confidence, and the lifter has no way to check the guess. We propose a lifter that can look back at the image. Each joint, in each frame, may consult a small window of image features centred on its own predicted 2D location. The cost grows linearly with the number of frames, not quadratically. The crops it reads keep their true shape, and image positions are encoded as camera rays, so squashing a tall hand crop into a square never quietly changes the geometry.
The bet is that most of the damage done by occlusion in two-stage pipelines comes from the lifter being blind, not from the lifter being weak. A few dozen well-placed image tokens per joint should recover most of the accuracy of dense image-plus-video attention, at a small fraction of the compute. The proposal is built so that a cheaper explanation can win if it is the true one. If simply passing the detector's confidence scores to the lifter closes the gap, the experiments will show it.
Core Claim
Two literatures are stuck on opposite sides of a trade-off. Coordinate-only video lifters such as PoseFormer treat each 2D joint as a token and attend across up to 81 frames. They are fast, and the temporal context smooths out per-frame noise. But they throw away pixels, so a confidently misplaced occluded joint passes straight into 3D. GATOR improves the structure of the lifter by writing skeleton distances into the attention matrix, but it still runs on coordinates alone and on single frames. Its 31.2 mm PA-MPJPE (Procrustes-aligned mean per-joint position error) on Human3.6M is measured with ground-truth 2D input, which is exactly the setting where the cascade problem disappears. Invariant Teacher and Equivariant Student and View Invariant 3D Human Pose Estimation motivate this proposal: both report that lifters degrade sharply once detected 2D points replace ground truth, and that the lost pixel context cannot be recovered downstream.
On the other side, HandOccNet shows that image features do help under hand–object occlusion. It splits the feature map into visible and occluded regions and uses attention to copy features from visible areas into hidden ones, reaching a 9.1 mm mean joint error on HO-3D. But it works on single 256×256 square crops and uses full self-attention over the whole feature map. It has no temporal reasoning and no route to long videos.
Our claim is specific. A lifter whose joint tokens cross-attend only to $k$ image tokens sampled around their own predicted 2D locations should beat both families under mutual occlusion on HO-3D and InterHand2.6M. With $J$ joints and $T$ frames, its attention cost scales as
$$C_{\text{ours}} = O\big(J T k + (J T)\, T\big) \quad \text{versus} \quad C_{\text{dense}} = O\big((T N)^2\big).$$
Here $N$ is the number of image tokens per frame. The first term counts the local lookups. The second counts temporal attention among joint tokens, which follows PoseFormer's factorisation. Because $J$ is 21 per hand and we use $k$ of about 16–49, the image term is small even at 81 frames.
Positions are encoded as camera rays rather than pixel indices. For each sampled pixel location $(u,v)$ in the original, uncropped image with camera intrinsics matrix $K$, the token receives the embedding of the unit ray
$$\mathbf{r} = \frac{K^{-1}[u, v, 1]^\top}{\lVert K^{-1}[u, v, 1]^\top \rVert}.$$
This makes the geometry the same whether a crop was kept at its native shape, letterboxed, or warped. Warping changes the image appearance but not the rays. Separating the two effects lets us measure, for the first time on hand benchmarks, how much of the 2D error comes from anisotropic resizing itself.
Research Program
The first piece of work is diagnostic. Hand crops in HO-3D and InterHand2.6M are frequently elongated; InterHand2.6M frames are natively 512×334. We will retrain a standard 2D hand keypoint detector three ways: on warped square crops, on letterboxed crops, and on native-aspect crops tokenised in the style of NaViT, which packs variable-length patch sequences and uses factorised x/y position embeddings. The token budget will be the same across all three. We will then plot 2D joint error, and the resulting 3D error, against crop elongation and against occlusion level. From the same runs we will measure an error amplification ratio: the slope of 3D error against 2D error for each lifter. This is the quantity the proposal claims to reduce.
The second piece is the model. A frozen or lightly fine-tuned image backbone produces native-aspect feature maps for each frame. A joint token starts from its 2D coordinate and detector confidence. In each layer it first runs a local cross-attention: it samples $k$ features on a small grid around its current 2D estimate, with ray embeddings attached, and attends over them. This is the "look back" step. Within each frame, joint tokens then attend to each other with GATOR-style hop-distance encodings, extended across the two hands or across hand and object for interacting scenes. Finally, each joint attends to itself across frames, as in PoseFormer. The 2D estimate is updated after every layer, so later lookups can move to where the evidence actually is.
That update step is the mechanism we care about most. It gives the lifter a way to override the detector rather than merely reweight it. A sigmoid confidence gate, borrowed from HandOccNet's dual attention, lets a joint ignore its window when the window is uninformative. This matters when the joint is occluded and the visible evidence lies on a neighbouring finger, which intra-frame attention supplies instead.
Contact and penetration terms in the style of HOLD and HO-3D_v3 are deliberately not part of the core. They are added only as an optional training loss, so that any gain can be attributed to the pixel lookup rather than to physics priors.
Baselines
The naive reference point is PoseFormer, retrained on 2D detections from the same detector as ours, on the same splits and with the same number of frames. This is the coordinate-only video lifter in its cleanest form. Beating it shows that looking back at pixels helps at all. No reported result found on Lacuna for PoseFormer on HO-3D or InterHand2.6M; its reported 44.3 mm MPJPE on Human3.6M is not comparable.
The strongest image-based competitor is HandOccNet. It is run from its official code on our exact HO-3D split, retrained with the same backbone family and the same tuning budget (we will match its learning-rate search to ours), and evaluated per frame. Its reported figure is a 9.1 mm mean joint error on HO-3D (HandOccNet), with 10.8 mm 3D joint error on FPHA. We will re-establish both numbers in our pipeline before comparing. Beating it means lower joint error on the occluded-joint subset at no more than half its attention FLOPs per frame. Matching it overall while using temporal context alone would not count.
For interacting hands, the reference is InterNet from InterHand2.6M: 16.02 mm MPJPE on interacting hands and 32.57 mm relative root error. We will report against its protocol and also rerun it on detections from our detector.
We also need a strong dense upper reference. This is a ViViT Model-2-style factorised encoder, from ViViT, run over full HandOccNet-style feature maps across the same frames. It tells us how much accuracy the local lookup gives up relative to dense spatiotemporal attention, and at what compute cost.
Three ablations isolate the key ingredient. The first removes the 2D update step, so lookups stay at the detector's original guess. The second replaces image tokens with the detector's own heatmap patch at the same location. The third replaces ray embeddings with pixel-index embeddings, under each of the three crop modes. The second ablation is the decisive one.
Why not just…?
Why not just give the lifter the detector's confidence and heatmaps instead of pixels? This is the real threat. If the cascade failure is mostly about the lifter not knowing which joints to distrust, a confidence-aware PoseFormer captures that cheaply. Our heatmap-patch ablation is exactly this alternative, run at matched budget. If it comes within about 10% of our occluded-joint error, we will report that pixel re-entry is not worth its cost. The idea would then reduce to uncertainty-aware lifting. Our prior is that heatmaps encode only where the detector looked, not the object or the neighbouring finger that explains the occlusion. HandOccNet's gains from copying visible features into hidden regions support that prior, but do not prove it in the video setting.
Why not just letterbox the crop and correct the intrinsics? For the geometric part, this may be fully sufficient, and the ray encoding makes letterboxing and native-aspect crops geometrically equivalent by construction. The open question is token efficiency and fine detail. NaViT reports that padded batches waste large fractions of tokens, and LLaVA-UHD notes that padding reduces the resolution available to the object. The three-way crop experiment at a fixed token budget settles it. If letterboxing ties native-aspect crops, we will adopt letterboxing and drop the packing machinery.
Why not just extend HandOccNet over time with factorised dense attention? It would probably be accurate. The dense ViViT-style reference is this alternative. The question is the accuracy-per-FLOP curve over clip lengths from 16 to 81 frames. If dense attention dominates even at 81 frames, the case for local lookup collapses to a deployment argument, and we will say so.
Why not just use per-sequence optimisation like HOLD? HOLD's 24.2 mm hand MPJPE comes from hours of per-video optimisation. In-Hand 3D Object Reconstruction reports about 14 hours per sequence. Both depend on an upstream pose estimate that can fail under occlusion. We position our model as a better initialiser for such methods, not a replacement, and will test that by feeding our poses into a HOLD-style refinement.
Proposed Experiments
Training and evaluation use HO-3D_v3, with 83,325 training and 20,137 test images; Dex-YCB, for which there is no reported numerical result on Lacuna; InterHand2.6M; and FPHA for cross-dataset transfer.
Metrics are MPJPE and PA-MPJPE, reported overall and separately on joints labelled occluded. For two-hand scenes we add relative root error. We also report the error amplification ratio, attention FLOPs, and wall-clock time per clip.
The decisive checks are four. First, our model must reduce the error amplification ratio relative to PoseFormer when both run on identical detections. Second, it must beat the heatmap-patch ablation on occluded joints by a clear margin. Third, it must approach the dense factorised reference to within a small gap at a fraction of its compute as clips lengthen. Fourth, the crop-mode study must show whether warping causes measurable 2D and 3D error once ray encoding removes the geometric confound.
Each check has a stated failure outcome, and each failure changes what we would claim.
References
- 3D Human Pose Estimation with Spatial and Temporal Transformers
- Gator: Graph-Aware Transformer with Motion-Disentangled Regression for Human Mesh Recovery from a 2D Pose
- Invariant Teacher and Equivariant Student for Unsupervised 3D Human Pose Estimation
- View Invariant 3D Human Pose Estimation
- HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network
- Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
- HOLD: Category-Agnostic 3D Reconstruction of Interacting Hands and Objects from Video
- HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset
- InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image
- ViViT: A Video Vision Transformer
- LLaVA-UHD: An LMM Perceiving Any Aspect Ratio and High-Resolution Images
- In-Hand 3D Object Reconstruction from a Monocular RGB Video