Force-Aware Interface via Electromyography for Natural VR/AR Interaction

Yunxiang Zhang, Benjamin Liang, Boyuan Chen, Paul M. Torrens, Seyed Farokh Atashzar, Dahua Lin, Qi Sun

Abstract

Current virtual and augmented reality (VR/AR) systems primarily rely on positional tracking, often neglecting the force of interaction. This paper presents a novel neural interface that utilizes non-invasive surface electromyography (sEMG) to predict individual finger forces in real-time. By transforming muscle electrical activity into virtual physics, the system enables users to interact with digital environments using natural, bare-handed force, significantly enhancing the perception of material stiffness without the need for bulky haptic hardware.

Introduction

While modern VR/AR has achieved high fidelity in visual and auditory domains, the "feel" of digital worlds remains a significant challenge. Most systems track where a finger is but ignore how hard it is pressing. Closing this loop requires a system where user-applied force results in physically accurate virtual reactions. Existing force-sensing solutions typically involve cumbersome haptic gloves or vision-based systems that struggle with subtle muscle tensions. This research addresses these limitations by using sEMG sensors on the forearm to decode finger forces directly from muscle signals.

Overview of pinch and push interactions using the sEMG armband for real-time virtual object manipulation.

Methodology

The researchers developed an end-to-end pipeline to translate raw muscle activity into real-time virtual deformation:

  1. Hardware and Data Acquisition: Eight wireless sEMG sensors are placed around the upper forearm. A new EMG-Force dataset was collected from 9 participants performing 11 hand-object interactions, using a high-resolution pressure sensor for ground truth.
  2. Signal Processing: Raw electrical signals are converted into 2D spectrograms via Short-Time Fourier Transform (STFT). This approach treats the noisy 1D data as an image, allowing the use of advanced computer vision techniques.
  3. Neural Architecture: A lightweight Convolutional Neural Network (CNN) with an encoder-decoder structure was designed for low-latency performance. A unique "joint classification-regression" loss function ensures the model first identifies if a finger is active before estimating the specific force, reducing input jitter.
  4. Transfer Learning: To account for physiological differences between users, the system employs a brief 2-minute calibration process that adapts a pre-trained model to new individuals.

Results and Key Insights

The system demonstrates high precision and responsiveness, achieving a mean error (NRMSE) of just 3.3% in force estimation with a total latency of 18.7ms—well within the requirements for 90Hz VR headsets.

Demonstration of force-aware interaction where the system predicts a 30N downward force, resulting in realistic deformation of virtual objects.

Key findings include:

  • Enhanced Perception: In psychophysical studies, users' ability to distinguish between different material stiffnesses improved by over 60% when force-aware feedback was enabled compared to traditional position-based methods.
  • Ubiquitous Control: The system can identify specific finger "clicks" on flat surfaces with 92.7% recall, potentially turning any surface into a multi-touch interface.
  • Robustness: The model remains effective even with minor sensor shifts, a common issue with wearable armbands.

Discussion and Future Work

While the current system provides a robust foundation for bare-handed force interaction, it currently relies on training data from flat surfaces. Future research will focus on expanding the model to handle complex 3D grasping of irregular objects. Additionally, the researchers aim to integrate sEMG force sensing with optical hand tracking to create a hybrid system that captures both high-fidelity hand pose and precise exertion levels, moving closer to a zero-calibration, consumer-ready interface.

Summary updated

Abstract

While tremendous advances in visual and auditory realism have been made for virtual and augmented reality (VR/AR), introducing a plausible sense of physicality into the virtual world remains challenging. Closing the gap between real-world physicality and immersive virtual experience requires a closed interaction loop: applying user-exerted physical forces to the virtual environment and generating haptic sensations back to the users. However, existing VR/AR solutions either completely ignore the force inputs from the users or rely on obtrusive sensing devices that compromise user experience. By identifying users' muscle activation patterns while engaging in VR/AR, we design a learning-based neural interface for natural and intuitive force inputs. Specifically, we show that lightweight electromyography sensors, resting non-invasively on users' forearm skin, inform and establish a robust understanding of their complex hand activities. Fuelled by a neural-network-based model, our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. Through an interactive psychophysical study, we show that human perception of virtual objects' physical properties, such as stiffness, can be significantly enhanced by our interface. We further demonstrate that our interface enables ubiquitous control via finger tapping. Ultimately, we envision our findings to push forward research towards more realistic physicality in future VR/AR.
BibTeX
@article{zhang2022force,
  author = {Yunxiang Zhang and Benjamin Liang and Boyuan Chen and Paul M. Torrens and Seyed Farokh Atashzar and Dahua Lin and Qi Sun},
  title = {{Force-Aware Interface via Electromyography for Natural VR/AR Interaction}},
  journal = {ACM Trans. Graphics},
  year = {2022},
  doi = {10.1145/3550454.3555461},
}
Figure