Virtual reality interactions via a user-generic ultrasound human-machine interface for wrist and hand tracking

Bruno Grandi Sgambato, Bálint K. Hodossy, Deren Y. Barsakcioglu, Xingchen Yang, Anette Jakob, Marc Fournelle, Meng-Xing Tang, Dario FarinaNature Communications 2025 · DOI

Abstract

As computers move from desktop screens into our glasses, traditional controllers such as keyboards and mice have proven impractical. A control interface for immersive experiences needs to seamlessly transport intention from the real to the virtual world while remaining portable, accurate, and robust. Here, we present an easy-to-wear, dry-contact and portable ultrasound armband that can decode morphological information and act as a virtual reality controller by predicting hand and wrist kinematics. Using our armband, we collected a large dataset of paired ultrasound and hand kinematics and used it to train supervised deep-learning models capable of predicting hand kinematics from ultrasound. We explored how diverse intra-session, cross-session, and cross-participant data shifts affect model performance. Further, we proposed methods for data conditioning, augmentation, and a referencing strategy to mitigate the influence of confounding factors and to achieve accurate prediction of hand kinematics on unseen users without fine-tuning. Finally, we demonstrated the feasibility of our interface in a real-time virtual reality control framework. Using the developed ultrasound interface, participants completed challenging interaction tasks with simulated contact physics. This work demonstrates the potential of ultrasound-based technologies as a virtual reality interface, showcasing strong performance, robustness, and generalization potential.

Introduction

As computing shifts from desktop environments to immersive wearable glasses (XR/VR/AR), the need for intuitive, portable, and robust human-machine interfaces (HMIs) has never been greater. Traditional input devices like keyboards and mice are impractical in these settings, while handheld controllers often break immersion. Current wearable alternatives, such as optical tracking and surface electromyography (sEMG), face significant challenges including occlusion and signal instability due to skin conditions or sensor placement.

Researchers have developed a "plug-and-play" ultrasound armband that addresses these hurdles. By treating multi-channel ultrasound signals as morphological data and employing a novel "referencing" strategy, the system achieves high-accuracy, zero-shot generalization across different users and sessions without requiring user-specific calibration.

Methods and System Design

The core of the interface is a modular, 32-channel A-mode ultrasound bracelet. Unlike bulky clinical B-mode probes, A-mode uses simple, single-element transducers that are low-power and suitable for long-term wear.

The 32-channel A-mode ultrasound bracelet designed for portable hand and wrist tracking.

Key technical innovations include:

  1. Multi-Receive Image Formation: The system captures complex geometric relationships by having multiple sensors listen to the echo of a single transmitting sensor, "stitching" these signals into 2D images.
  2. ConvNeXt Architecture: A state-of-the-art convolutional neural network processes these ultrasound "images" to learn spatial patterns of muscle deformation.
  3. Domain-Specific Augmentation: Custom techniques simulate bracelet rotation and signal shifts during training to ensure robustness against physical sensor displacement.
  4. Referencing Strategy: To achieve user-generic performance, the model compares a "target" frame of current movement against a "reference" frame of the user's arm in a neutral pose. This cancels out individual anatomical differences.

Key Insights: Ultrasound as a Morphological Interface

Unlike sEMG, which measures electrical intent, ultrasound measures the physical result of muscle contraction—shape change. This "morphological" signal is inherently more stable and directly correlated to kinematics.

Multi-channel A-mode ultrasound signals representing the internal muscle morphology during hand movements.

The researchers collected a dataset of approximately 24 hours (1.1 million samples) involving 10 initial participants for model development and 16 participants (including 8 new users) for real-time validation. This diversity allowed the model to learn to ignore "real-world" noise such as arm position changes and bracelet rotation.

Results and Real-Time Validation

The system demonstrated exceptional performance:

  • High Accuracy: Median $R^2$ values exceeded 0.9 for wrist rotation and hand open/close gestures on unseen users.
  • Zero-Shot Generalization: The referencing strategy allowed the model to perform on new users at a level comparable to models trained on significantly larger, user-specific datasets.
  • VR Integration: In real-time tests, participants successfully completed complex tasks in physics-based simulations, such as pouring beads and operating door handles, with no prior training.
  • Robustness: The system maintained tracking even when the user's hand was obscured, overcoming the primary limitation of vision-based systems like Leap Motion.

Limitations and Future Outlook

While the system excels at primary wrist and hand movements, performance for radial-ulnar deviation (side-to-side waving) remains an area for improvement, potentially requiring higher frequency transducers. Future work will focus on shrinking the external driving electronics into a fully self-contained, battery-powered wearable and expanding the training set to include more diverse anatomical profiles.

Summary updated

Abstract

As computers move from desktop screens into our glasses, traditional controllers such as keyboards and mice have proven impractical. A control interface for immersive experiences needs to seamlessly transport intention from the real to the virtual world while remaining portable, accurate, and robust. Here, we present an easy-to-wear, dry-contact and portable ultrasound armband that can decode morphological information and act as a virtual reality controller by predicting hand and wrist kinematics. Using our armband, we collected a large dataset of paired ultrasound and hand kinematics and used it to train supervised deep-learning models capable of predicting hand kinematics from ultrasound. We explored how diverse intra-session, cross-session, and cross-participant data shifts affect model performance. Further, we proposed methods for data conditioning, augmentation, and a referencing strategy to mitigate the influence of confounding factors and to achieve accurate prediction of hand kinematics on unseen users without fine-tuning. Finally, we demonstrated the feasibility of our interface in a real-time virtual reality control framework. Using the developed ultrasound interface, participants completed challenging interaction tasks with simulated contact physics. This work demonstrates the potential of ultrasound-based technologies as a virtual reality interface, showcasing strong performance, robustness, and generalization potential.
BibTeX
@inproceedings{sgambato2025virtual,
  author = {Bruno Grandi Sgambato and Bálint K. Hodossy and Deren Y. Barsakcioglu and Xingchen Yang and Anette Jakob and Marc Fournelle and Meng-Xing Tang and Dario Farina},
  title = {{Virtual reality interactions via a user-generic ultrasound human-machine interface for wrist and hand tracking}},
  booktitle = {Nature Communications},
  year = {2025},
  doi = {10.1038/s41467-025-66001-6},
}
Figure