Towards Predicting Fine Finger Motions from Ultrasound Images via Kinematic Representation

Dean Zadok, Oren Salzman, Alon Wolf, Alex M. Bronstein

Abstract

Traditional prosthetic control often relies on surface electromyography (EMG), which lacks the resolution for fine, dexterous tasks. This paper presents an end-to-end system that utilizes ultrasound (US) imaging of forearm muscles to predict continuous finger movements. By introducing an intermediate kinematic representation—modeling the hand as a robotic manipulator—the proposed Configuration-Based Multi-Frame (CBMF) model achieves high precision in complex tasks like piano playing and typing.

Introduction

Current robotic prostheses are primarily limited to coarse gestures, such as simple grasping or pointing. While Electromyography (EMG) is the standard for non-invasive control, its signals are inherently noisy and struggle to differentiate the overlapping muscle activations required for high-fidelity tasks.

Ultrasound imaging offers a high-resolution alternative by capturing the physical deformation of deep muscle tissues. However, most US-based research has focused on classifying static gestures. This work addresses the challenge of continuous, fine-grained motion inference: predicting exactly which finger is moving and its specific trajectory based on a sequence of ultrasound frames.

Methods

The researchers propose a deep learning framework that treats the human hand's anatomy as a robotic system. The core innovation is the Configuration-Based Multi-Frame (CBMF) model, an auto-encoder architecture designed to inject anatomical domain knowledge into the learning process.

The CBMF architecture maps ultrasound sequences to a kinematic hand representation before decoding finger actions.

The CBMF model consists of:

  • Encoder ($h_\theta^e$): A network that maps a sequence of ultrasound images to an intermediate kinematic representation, consisting of 17 joint angles of the hand and wrist.
  • Decoder ($h_\phi^d$): A network that uses this skeletal configuration, along with latent features, to predict the final finger-pressing actions.

This approach was compared against a baseline Multi-Frame (MF) model, which directly predicts pressing vectors from US sequences without the intermediate kinematic step.

Experimental Setup

Data was collected from 12 healthy subjects performing two tasks: piano playing and keyboard typing. Ground-truth joint angles were recorded using a Vicon motion-capture system, while muscle deformations were captured via a wireless B-mode ultrasound probe on the forearm.

Anatomical labeling of the forearm ultrasound (left) and the corresponding kinematic joint model (right).

The dataset includes over 86,000 samples. The ultrasound images capture several key muscle groups, including the Flexor Digitorum Superficialis (FDS) and Flexor Digitorum Profundus (FDP), which are responsible for finger flexion.

Key Results and Insights

The CBMF model significantly outperformed the baseline, particularly in complex scenarios.

  • Performance: For piano playing, the CBMF model achieved an F1-score of ~0.86. In the more challenging typing task, it reached ~0.69, whereas simpler models often failed to converge.
  • Kinematics as a Regularizer: The intermediate joint-angle prediction acts as a physical constraint, reducing false positives by ensuring predicted actions are anatomically consistent with muscle states.
  • Task Sensitivity: Piano playing was easier to predict than typing due to the greater "travel" distance of the keys (10.45 mm vs 2.95 mm), which produces more distinct muscle deformations.
  • Anatomical Bias: Accuracy was highest for the middle three fingers. The thumb and pinky muscles are less visible to a single transverse probe, suggesting a limitation of current sensor placement.

Conclusion and Future Work

This research demonstrates the feasibility of real-time, fine-grained robotic control via ultrasound. By re-framing the hand as a kinematic system, the model achieves the precision necessary for tasks like playing an instrument. Future work will focus on adapting the system for amputees, whose residual muscle morphology differs from healthy subjects, and improving robustness to sensor displacement.

Summary updated

BibTeX
@inproceedings{zadok2023towards,
  author = {Dean Zadok and Oren Salzman and Alon Wolf and Alex M. Bronstein},
  title = {{Towards Predicting Fine Finger Motions from Ultrasound Images via Kinematic Representation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160601},
}
Figure