Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics

Matteo Cognolato, Arjan Gijsberts, Valentina Gregori, Gianluca Saetta, Katia Giacomino, Anne-Gabrielle Mittaz Hager, Andrea Gigli, Diego Faccio, Cesare Tiengo, Franco Bassetto, Barbara Caputo, Peter Brugger, Manfredo Atzori, Henning Müller

Abstract

A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person's life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics.

Introduction

Current state-of-the-art hand prostheses typically rely on surface electromyography (sEMG)—detecting the electrical activity of residual limb muscles to trigger movements. While functional, sEMG is notoriously "noisy." Signal quality fluctuates based on electrode sweat, muscle fatigue, and even the physical position of the arm. This makes fine-grained control of dexterous robotic hands frustratingly unreliable in daily life.

However, humans don't move in a vacuum. We almost always look at an object before we grab it. This "eye-hand coordination" provides a massive amount of contextual information (e.g., if you are looking at a coffee mug handle, you likely intend to use a "lateral" or "tripod" grasp). The challenge has been a lack of high-quality, synchronized data that combines these internal muscle signals with external visual and gaze data, especially from the target demographic: individuals with limb loss.

Methods and Experimental Setup

The researchers developed a rigorous acquisition setup to synchronize five distinct data streams: sEMG (12 electrodes), forearm accelerometry, gaze tracking (2D and 3D), first-person video, and head inertial data (IMU).

The experimental setup includes eye-tracking glasses, sEMG electrodes on the forearm, and synchronized data acquisition for real-time monitoring.

The protocol involved 10 common grasp types performed on 18 different household objects. Crucially, the study included both static tasks (grasping without moving) and dynamic tasks (functional movements like drinking from a can or turning a key). To improve the dataset's utility for machine learning, the authors used a log-likelihood optimization method to "relabel" the data, adjusting timestamps to match when muscles actually activated rather than just when the instruction was given.

The MeganePro Dataset (MDS1)

The paper introduces the MeganePro dataset 1 (MDS1), which is publicly available on the Harvard Dataverse. It builds on the methodology of the well-known NinaPro project but adds the critical dimensions of gaze and vision.

Demographic and clinical information for the 15 transradial amputees and 30 able-bodied subjects included in the MDS1 dataset.

The dataset is unique because it allows for a direct, head-to-head comparison between amputee and able-bodied gaze-and-grasp behavior in identical experimental settings. Analysis confirms that gaze is an "amputation-resistant" signal; while muscle signals degrade due to atrophy or nerve reorganization after an injury, gaze behavior remains largely intact, acting as a stable "anchor" for control systems.

Key Results and Validation

To validate the data quality, the authors performed baseline classification tasks:

  • sEMG Classification: Using standard machine learning (LDA, k-NN, and KRLS), they achieved 63% to 82% accuracy in identifying the 10 grasp types from muscle signals alone. This serves as a benchmark for future researchers to beat by incorporating gaze and video data.
  • Gaze Precision: Technical validation showed that gaze tracking was highly accurate in the center of the field of view, with errors typically under 10–30 pixels—sufficient to identify which household object a user is targeting.
  • The "Pre-Grasp" Window: The data confirms that subjects fixated on the target object in 95.9% of trials. This confirms that gaze can act as a "lead" signal, giving a prosthetic controller a few hundred milliseconds of head-start to prepare the correct grasp pattern.

Discussion and Future Directions

While the dataset provides a robust foundation, real-world hurdles remain. Hardware issues, such as unreliable electrodes or physiological constraints affecting gaze tracking (e.g., strabismus), highlight the need for robust computer vision algorithms that can handle imperfect data. The next major step is "closing the loop"—using this multimodal data to drive a physical prosthetic in real-time to see if the theoretical accuracy gains translate to a better user experience.

Reference: Cognolato, M., et al. (2020). Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics. Scientific Data. https://doi.org/10.1038/s41597-020-0380-3

Summary updated

Abstract

A hand amputation is a highly disabling event, having severe physical and psychological repercussions on a person's life. Despite extensive efforts devoted to restoring the missing functionality via dexterous myoelectric hand prostheses, natural and robust control usable in everyday life is still challenging. Novel techniques have been proposed to overcome the current limitations, among them the fusion of surface electromyography with other sources of contextual information. We present a dataset to investigate the inclusion of eye tracking and first person video to provide more stable intent recognition for prosthetic control. This multimodal dataset contains surface electromyography and accelerometry of the forearm, and gaze, first person video, and inertial measurements of the head recorded from 15 transradial amputees and 30 able-bodied subjects performing grasping tasks. Besides the intended application for upper-limb prosthetics, we also foresee uses for this dataset to study eye-hand coordination in the context of psychophysics, neuroscience, and assistive robotics.
BibTeX
@inproceedings{cognolato2020gaze,
  author = {Matteo Cognolato and Arjan Gijsberts and Valentina Gregori and Gianluca Saetta and Katia Giacomino and Anne-Gabrielle Mittaz Hager and Andrea Gigli and Diego Faccio and Cesare Tiengo and Franco Bassetto and Barbara Caputo and Peter Brugger and Manfredo Atzori and Henning Müller},
  title = {{Gaze, visual, myoelectric, and inertial data of grasps for intelligent prosthetics}},
  booktitle = {Scientific},
  year = {2020},
  doi = {10.1038/s41597-020-0380-3},
}
Figure