Abstract
Traditional prosthetic control often relies on surface electromyography (EMG), which lacks the resolution for fine, dexterous tasks. This paper presents an end-to-end system that utilizes ultrasound (US) imaging of forearm muscles to predict continuous finger movements. By introducing an intermediate kinematic representation—modeling the hand as a robotic manipulator—the proposed Configuration-Based Multi-Frame (CBMF) model achieves high precision in complex tasks like piano playing and typing.
Introduction
Current robotic prostheses are primarily limited to coarse gestures, such as simple grasping or pointing. While Electromyography (EMG) is the standard for non-invasive control, its signals are inherently noisy and struggle to differentiate the overlapping muscle activations required for high-fidelity tasks.
Ultrasound imaging offers a high-resolution alternative by capturing the physical deformation of deep muscle tissues. However, most US-based research has focused on classifying static gestures. This work addresses the challenge of continuous, fine-grained motion inference: predicting exactly which finger is moving and its specific trajectory based on a sequence of ultrasound frames.
Methods
The researchers propose a deep learning framework that treats the human hand's anatomy as a robotic system. The core innovation is the Configuration-Based Multi-Frame (CBMF) model, an auto-encoder architecture designed to inject anatomical domain knowledge into the learning process.
The CBMF model consists of:
- Encoder ($h_\theta^e$): A network that maps a sequence of ultrasound images to an intermediate kinematic representation, consisting of 17 joint angles of the hand and wrist.
- Decoder ($h_\phi^d$): A network that uses this skeletal configuration, along with latent features, to predict the final finger-pressing actions.
This approach was compared against a baseline Multi-Frame (MF) model, which directly predicts pressing vectors from US sequences without the intermediate kinematic step.
Experimental Setup
Data was collected from 12 healthy subjects performing two tasks: piano playing and keyboard typing. Ground-truth joint angles were recorded using a Vicon motion-capture system, while muscle deformations were captured via a wireless B-mode ultrasound probe on the forearm.
The dataset includes over 86,000 samples. The ultrasound images capture several key muscle groups, including the Flexor Digitorum Superficialis (FDS) and Flexor Digitorum Profundus (FDP), which are responsible for finger flexion.
Key Results and Insights
The CBMF model significantly outperformed the baseline, particularly in complex scenarios.
- Performance: For piano playing, the CBMF model achieved an F1-score of ~0.86. In the more challenging typing task, it reached ~0.69, whereas simpler models often failed to converge.
- Kinematics as a Regularizer: The intermediate joint-angle prediction acts as a physical constraint, reducing false positives by ensuring predicted actions are anatomically consistent with muscle states.
- Task Sensitivity: Piano playing was easier to predict than typing due to the greater "travel" distance of the keys (10.45 mm vs 2.95 mm), which produces more distinct muscle deformations.
- Anatomical Bias: Accuracy was highest for the middle three fingers. The thumb and pinky muscles are less visible to a single transverse probe, suggesting a limitation of current sensor placement.
Conclusion and Future Work
This research demonstrates the feasibility of real-time, fine-grained robotic control via ultrasound. By re-framing the hand as a kinematic system, the model achieves the precision necessary for tasks like playing an instrument. Future work will focus on adapting the system for amputees, whose residual muscle morphology differs from healthy subjects, and improving robustness to sensor displacement.
Summary updated