DoF-Query Transformer for Adaptive Fusion of Egocentric Vision and Forearm HD-sEMG in Continuous Hand Kinematics

Jiahang Liu, Wenbo Zhang, Junsong Wang, Zhenzhi Ying, Siqi Hou, Naohiko Sugita, Liming ShuarXiv 2026 · arXiv

This paper presents a novel approach to tracking hand movements by combining what we see (computer vision) with what our muscles are doing (electromyography). By using a specialized Transformer architecture, the researchers have created a system that can accurately estimate 20 different joint angles of the hand, even when the hand is partially hidden from view.

TL;DR

Researchers developed a multimodal Transformer framework that fuses head-mounted camera footage with forearm muscle signals (HD-sEMG) to track 20 hand joint angles continuously. By using a novel "DoF-query" mechanism, the system adaptively weights each sensor based on the current context, significantly improving accuracy and providing high robustness to visual occlusions.

What Problem Does This Paper Address?

Accurate hand tracking is the "holy grail" for intuitive human-robot interaction, VR/AR, and teaching robots how to manipulate objects. However, current methods have a "Goldilocks" problem:

  • Vision-only systems (like a camera on a VR headset) are convenient but fail when fingers overlap, move too fast, or are hidden behind the palm (self-occlusion).
  • Wearable gloves provide great data but are bulky and interfere with the very thing we want to measure: natural touch and manipulation.
  • Standard EMG (muscle sensors) can "see" through occlusions but lacks the geometric precision of vision.

The challenge is fusion. How do you combine a 2D image with 128 channels of electrical muscle signals in a way that knows which sensor to trust at any given millisecond?

Methods

The authors propose a "finger-unencumbered" framework, meaning the fingers remain completely free of sensors. The architecture consists of three main parts:

  1. Vision Branch: Uses a Vision Transformer (ViT-H) backbone to extract spatial features from egocentric (first-person) RGB images.
  2. EMG Branch: Processes 128-channel high-density surface EMG (HD-sEMG) from the forearm. It uses a Transformer encoder to capture how muscle signals change over time and across different muscle groups.
  3. DoF-Query Transformer Decoder: This is the core innovation. Inspired by object detection models like DETR, the researchers assigned a "learnable query" to each of the 20 degrees of freedom (DoF) in the hand.

Instead of just mashing the vision and EMG data together, each joint "asks" the data: "Based on what's happening right now, which features are most relevant to my specific angle?" This allows the model to perform context-adaptive fusion—for example, relying more on EMG for a finger that is currently hidden from the camera.

Datasets

The researchers collected a new synchronized multimodal dataset:

  • Participants: 10 healthy adults.
  • Inputs: 30Hz egocentric RGB video and 2000Hz 128-channel HD-sEMG.
  • Ground Truth: High-precision 3D trajectories from a 10-camera Vicon motion capture system.
  • Protocol: Included 26 predefined gestures and 40 trials of unconstrained "free movement" per person to ensure the model could handle real-world variety.

Key Insights and Ideas

  • The Power of Specificity: Global fusion (treating the whole hand as one block of data) is suboptimal. By giving each joint its own "query," the model can learn that the thumb's movement might be better explained by specific muscle groups, while the index finger's position is easily seen by the camera.
  • Implicit Reliability: The attention mechanism effectively acts as a dynamic weighting system. When the hand enters a "visually ambiguous" state (like a side view where fingers overlap), the model automatically shifts its attention mass toward the EMG tokens.
  • Complementary Strengths: Vision provides the "absolute" geometry of the hand, while EMG provides the "intent" and "activation" that persists even when the hand is tucked behind an object.

Key Results

The fusion model significantly outperformed unimodal baselines across the board:

  • Accuracy: Achieved a mean $R^2$ of 0.88 and a Mean Absolute Error (MAE) of 2.80°. For comparison, vision-only achieved 0.82 (3.21°) and EMG-only achieved 0.68 (5.13°).
  • Occlusion Robustness: This is where the model shined. In synthetic tests, when 30% of the hand image was occluded, the vision-only model's performance collapsed ($R^2$ dropped near zero). The fusion model, however, maintained high accuracy by leaning on the EMG signals.
  • Joint-Wise Gains: The biggest improvements were seen in distal joints (fingertips) and the thumb, which are historically the hardest to track with vision alone due to frequent self-occlusion.

Limitations and Future Work

  • Explicit Quality Assessment: While the model adaptively fuses data, it doesn't explicitly detect when a sensor is failing (e.g., a camera lens being covered). Under extreme occlusion (>50%), the "noisy" visual features can still slightly degrade the overall estimate compared to an EMG-only model.
  • Generalization: The current study focused on 10 participants in a controlled lab setting. Future work is needed to see how well these "muscle signatures" generalize across different people without extensive recalibration.
  • Real-time Deployment: While the paper demonstrates the framework's accuracy, the authors suggest that future work will focus on optimizing the Transformer for real-time, low-latency applications in robotics and AR.

Summary updated

Abstract

Continuous and accurate estimation of human hand kinematics is important not only for intuitive human-machine interaction, but also for capturing fine-grained human manipulation demonstrations that can provide action supervision for embodied-intelligence datasets and dexterous robot learning. However, individual sensing modalities may be insufficient for reliable estimation under varying operating conditions. We present a finger-unencumbered multimodal framework for continuous regression of 20 hand joint angles from egocentric RGB images and 128-channel forearm high-density surface electromyography (HD-sEMG). The framework combines a Vision Transformer-based visual encoder, a Transformer-based EMG encoder, and a degree of freedom (DoF) query Transformer fusion decoder. To account for the varying informativeness of visual and EMG cues across joint DoFs and input conditions, the decoder assigns a separate learnable query to each DoF, enabling DoF-specific and context-adaptive fusion of the two modalities. The framework was evaluated on synchronized multimodal data from 10 participants. Averaged across participants, the proposed fusion model achieved a mean coefficient of determination ($R^2$) of $0.88$ and a mean absolute error (MAE) of $2.80^\circ$, compared with $0.82$ and $3.21^\circ$ for the vision-only model and $0.68$ and $5.13^\circ$ for the EMG-only model. It also outperformed the evaluated representative unimodal baselines and alternative fusion designs. DoF-wise analyses showed larger fusion gains for visually ambiguous joints and viewing conditions, while synthetic visual occlusion demonstrated greater robustness than vision-only estimation. These results highlight the complementary value of egocentric vision and forearm HD-sEMG for capturing fine-grained human hand kinematics, with potential applications in human demonstration acquisition and dexterous robot learning.
BibTeX
@misc{liu2026dof,
  author = {Jiahang Liu and Wenbo Zhang and Junsong Wang and Zhenzhi Ying and Siqi Hou and Naohiko Sugita and Liming Shu},
  title = {{DoF-Query Transformer for Adaptive Fusion of Egocentric Vision and Forearm HD-sEMG in Continuous Hand Kinematics}},
  year = {2026},
  eprint = {2611.20196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20196},
}
Figure