Abstract
A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person's life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics.
Introduction
Current state-of-the-art hand prostheses typically rely on surface electromyography (sEMG)—detecting the electrical activity of residual limb muscles to trigger movements. While functional, sEMG is notoriously "noisy." Signal quality fluctuates based on electrode sweat, muscle fatigue, and even the physical position of the arm. This makes fine-grained control of dexterous robotic hands frustratingly unreliable in daily life.
However, humans don't move in a vacuum. We almost always look at an object before we grab it. This "eye-hand coordination" provides a massive amount of contextual information (e.g., if you are looking at a coffee mug handle, you likely intend to use a "lateral" or "tripod" grasp). The challenge has been a lack of high-quality, synchronized data that combines these internal muscle signals with external visual and gaze data, especially from the target demographic: individuals with limb loss.
Methods and Experimental Setup
The researchers developed a rigorous acquisition setup to synchronize five distinct data streams: sEMG (12 electrodes), forearm accelerometry, gaze tracking (2D and 3D), first-person video, and head inertial data (IMU).
The protocol involved 10 common grasp types performed on 18 different household objects. Crucially, the study included both static tasks (grasping without moving) and dynamic tasks (functional movements like drinking from a can or turning a key). To improve the dataset's utility for machine learning, the authors used a log-likelihood optimization method to "relabel" the data, adjusting timestamps to match when muscles actually activated rather than just when the instruction was given.
The MeganePro Dataset (MDS1)
The paper introduces the MeganePro dataset 1 (MDS1), which is publicly available on the Harvard Dataverse. It builds on the methodology of the well-known NinaPro project but adds the critical dimensions of gaze and vision.
The dataset is unique because it allows for a direct, head-to-head comparison between amputee and able-bodied gaze-and-grasp behavior in identical experimental settings. Analysis confirms that gaze is an "amputation-resistant" signal; while muscle signals degrade due to atrophy or nerve reorganization after an injury, gaze behavior remains largely intact, acting as a stable "anchor" for control systems.
Key Results and Validation
To validate the data quality, the authors performed baseline classification tasks:
- sEMG Classification: Using standard machine learning (LDA, k-NN, and KRLS), they achieved 63% to 82% accuracy in identifying the 10 grasp types from muscle signals alone. This serves as a benchmark for future researchers to beat by incorporating gaze and video data.
- Gaze Precision: Technical validation showed that gaze tracking was highly accurate in the center of the field of view, with errors typically under 10–30 pixels—sufficient to identify which household object a user is targeting.
- The "Pre-Grasp" Window: The data confirms that subjects fixated on the target object in 95.9% of trials. This confirms that gaze can act as a "lead" signal, giving a prosthetic controller a few hundred milliseconds of head-start to prepare the correct grasp pattern.
Discussion and Future Directions
While the dataset provides a robust foundation, real-world hurdles remain. Hardware issues, such as unreliable electrodes or physiological constraints affecting gaze tracking (e.g., strabismus), highlight the need for robust computer vision algorithms that can handle imperfect data. The next major step is "closing the loop"—using this multimodal data to drive a physical prosthetic in real-time to see if the theoretical accuracy gains translate to a better user experience.
Reference: Cognolato, M., et al. (2020). Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics. Scientific Data. https://doi.org/10.1038/s41597-020-0380-3
Summary updated