Abstract
With the advancement in computing and robotics, it is necessary to develop fluent and intuitive methods for interacting with digital systems, augmented/virtual reality (AR/VR) interfaces, and physical robotic systems. Hand motion recognition is widely used to enable these interactions. Hand configuration classification and MCP joint angle detection is important for a comprehensive reconstruction of hand motion. sEMG and other technologies have been used for the detection of hand motions. Forearm ultrasound images provide a musculoskeletal visualization that can be used to understand hand motion. Recent work has shown that these ultrasound images can be classified using machine learning to estimate discrete hand configurations. Estimating both hand configuration and MCP joint angles based on forearm ultrasound has not been addressed in the literature. In this paper, we propose a CNN based deep learning pipeline for predicting the MCP joint angles. The results for the hand configuration classification were compared by using different machine learning algorithms. SVC with different kernels, MLP, and the proposed CNN have been used to classify the ultrasound images into 11 hand configurations based on activities of daily living. Forearm ultrasound images were acquired from 6 subjects instructed to move their hands according to predefined hand configurations. Motion capture data was acquired to get the finger angles corresponding to the hand movements at different speeds. Average classification accuracy of 82.7% for the proposed CNN and over 80% for SVC for different kernels was observed on a subset of the dataset. An average RMSE of 7.35 degrees was obtained between the predicted and the true MCP joint angles. A low latency (6.25 - 9.1 Hz) pipeline has been proposed for estimating both MCP joint angles and hand configuration aimed at real-time control of human-machine interfaces.
1. Introduction
Interacting with digital systems, AR/VR environments, and robotic prosthetics requires precise hand-motion recognition. Traditionally, this is achieved using surface electromyography (sEMG), which measures electrical activity on the skin, or vision-based systems. However, both have significant drawbacks: sEMG is prone to signal noise and muscle fatigue, while cameras suffer from occlusion and lighting issues.
Forearm ultrasound—or sonomyography—offers a 2D visualization of actual muscle and tendon movements. While previous work used ultrasound for discrete gesture classification, this paper tackles the harder challenge of simultaneous estimation: identifying the hand's overall configuration while also tracking the specific angles of the metacarpophalangeal (MCP) joints (the "knuckles") in real-time.
2. Methodology
2.1 Data Acquisition
The study involved data collection from 6 subjects. B-mode ultrasound images of the right forearm were captured at 25 Hz. To establish ground truth, 3D positional data was acquired from a Vicon Nexus motion capture system with 16 markers attached to the hand. Subjects performed 11 hand configurations (e.g., fist, hook, index flex) at three different speeds (0.5 Hz, 1 Hz, and 2 Hz) to simulate various activities of daily living (ADLs).
2.2 Model Architectures
The authors proposed a dual-purpose deep learning pipeline:
- Hand Configuration Classification: Several models—Support Vector Classifiers (SVC), Multi-Layer Perceptrons (MLP), and a custom CNN—were compared to categorize ultrasound images into 11 distinct configurations.
- Joint Angle Estimation: A CNN architecture (inspired by a modified VGG16) was used for continuous regression to predict the MCP joint angles for the index, middle, ring, and pinky fingers. The network consists of five convolutional sections followed by dense layers.
2.3 Combined Real-Time Pipeline
To enable real-time use, a low-latency system was developed. It first uses a saved SVC model to identify the hand configuration and then selects a configuration-specific CNN model to estimate the joint angles. This entire process runs at a frequency of 6.25 to 9.1 Hz.
3. Key Results and Insights
- Accuracy: The proposed CNN achieved an average classification accuracy of 82.7% ± 9.7%. For joint angle estimation, the system achieved an average Root Mean Square Error (RMSE) of 7.35° ± 1.3°.
- Spatial Features: The work demonstrates that the spatial features in 2D B-mode images are critical for accurate joint angle prediction compared to 1D A-mode signals.
- Speed and Complexity: Models performed best at slower movement speeds (0.5 Hz). Configurations involving multiple fingers (like the "Fist") had higher error rates because complex, overlapping muscle movements are harder to disentangle. The pinky finger consistently showed the highest error due to its smaller muscle representation.
- Latency: The pipeline processing rate of up to 9.1 Hz is a significant step toward "on-the-fly" human-robot interaction.
4. Discussion and Future Work
While promising, the current system has limitations. The thumb is currently excluded because its primary movements are controlled by intrinsic muscles in the hand rather than the extrinsic muscles in the forearm. Additionally, the models struggle to generalize across different subjects due to anatomical differences.
Future work will focus on incorporating thumb tracking and transitioning from bulky research-grade ultrasound systems to wearable ultrasound patches—small, low-power devices that could bring this technology into daily life.
Summary updated