Abstract
Hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person’s life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among which the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics.
Introduction
Current myoelectric prostheses—artificial limbs controlled by electrical signals from remaining muscles—are notoriously "brittle." While they perform adequately in controlled laboratory settings, real-world application often leads to failure due to factors like sweat, electrode shifts, muscle fatigue, or changes in arm posture. The fundamental challenge lies in intent recognition. Relying solely on surface electromyography (sEMG) lacks the necessary environmental context. This research addresses the scarcity of high-quality, multimodal data combining muscle activity with "contextual" cues, specifically gaze behavior and first-person visual information.
Experimental Design and Methods
The researchers developed a rigorous acquisition protocol to capture synchronized multimodal data. The experimental setup integrated several key sensors to provide a holistic view of the user's intent and action.
Sensor Integration: * Muscle Activity: 12 Delsys Trigno wireless electrodes recorded sEMG and 3-axis accelerometry from the forearm. * Vision & Gaze: Tobii Pro Glasses 2 recorded gaze points, first-person Full HD video, and head movements via an Inertial Measurement Unit (IMU). * Tasks: Participants performed 10 different grasp types (e.g., tripod, power sphere) using 18 common household objects in both static and dynamic (functional) scenarios.
To ensure data quality, a "relabeling" algorithm using log-likelihood optimization was implemented to align ground-truth labels with the actual onset of muscle activation, correcting for human reaction time delays.
The MeganePro Dataset 1 (MDS1)
The resulting MeganePro dataset 1 (MDS1), hosted on Harvard Dataverse, provides a comprehensive library for training robust machine learning models. It builds upon the legacy of the NinaPro project by adding critical dimensions of gaze and first-person video.
The dataset includes 45 participants: 15 transradial amputees with varying levels of limb loss and 30 age/gender-matched able-bodied controls. Data is provided in a synchronized "standard" format (resampled to 1926 Hz) and an "auxiliary" format at original sampling rates to preserve signal integrity.
Key Insights and Results
Analysis of the dataset revealed several critical findings for the future of prosthetic control:
- Gaze Stability: Unlike sEMG signals, which can degrade post-amputation due to muscle atrophy, gaze behavior remains remarkably stable. Fixation durations were nearly identical between amputees and able-bodied subjects (~0.43 seconds), suggesting that control algorithms may be highly transferable.
- The Pre-Grasp Window: In 95.9% of trials, subjects visually fixated on an object before reaching for it. This "look-ahead" provides a vital window for a smart prosthesis to pre-configure its grip autonomously.
- Classification Performance: Using sEMG data alone, grasp classification accuracies reached 63% to 82% across various classifiers (LDA, k-NN, KRLS), significantly exceeding the 50% baseline.
- Gaze Precision: The eye-tracking system maintained a precision of 4.4 mm to 11.5 mm at typical manipulation distances, sufficient for distinguishing between small, adjacent objects.
Conclusion and Future Directions
While the dataset highlights challenges such as sensor reliability and calibration drift at the edges of the visual field, it provides a foundational resource for the next generation of "semi-autonomous" prosthetics. By fusing computer vision for object identification, gaze for target confirmation, and sEMG for action triggering, engineers can move closer to natural, robust prosthetic control.
Summary updated