A Multimodal Intention Detection Sensor Suite for Shared Autonomy of Upper-Limb Robotic Prostheses

Marcus Gardner, C. Sebastian Mancero Castillo, Samuel Wilson, Dario Farina, Etienne Burdet, Boo Cheong Khoo, S. Farokh Atashzar, Ravi VaidyanathanSensors 2020 · DOI

TL;DR

This paper introduces a novel multimodal sensor suite that fuses mechanomyography (MMG), computer vision, and inertial measurement units (IMUs) to predict user intent for grasping objects with upper-limb robotic prostheses. This "shared autonomy" approach aims to reduce cognitive load and improve the intuitiveness of prosthetic control, demonstrating high accuracy in real-time grasp classification for common household objects.

What Problem Does This Paper Address?

Controlling advanced robotic prostheses, especially for tasks requiring precise manipulation like grasping, is incredibly challenging. Current systems often rely on indirect control methods (like button presses or mode switching) or limited sequential control, which places a significant cognitive burden on the user. This can lead to frustration, fatigue, and even abandonment of the prosthesis. The core problem is creating a more intuitive and less demanding Human-Machine Interface (HMI) that allows users to control their prostheses with greater dexterity and naturalness, mimicking the capabilities of a biological limb.

Methods

The researchers developed a multimodal sensor suite integrated into a shared autonomy framework. The system comprises three key subsystems:

  • Muscle Activation Subsystem (MASS): Utilizes custom-made acoustic MMG sensors embedded in a compression sleeve. MMG measures the mechanical vibrations of muscles, offering a more robust signal than traditional EMG in certain conditions. The system filters and processes the MMG signal, comparing it against a threshold to detect intentional muscle activation. Crucially, it uses IMU data to differentiate genuine muscle activation from motion artifacts.
  • Computer Vision Subsystem (CVSS): Employs a low-cost, wide-angle camera mounted on the user's head. This system performs basic image processing (contrast correction, edge detection, smoothing) to identify objects of interest. It uses a K-nearest neighbor (KNN) classifier to recognize objects based on extracted features like perimeter, area, and aspect ratios.
  • Grasp Prediction Subsystem (GPSS): This is the core of the intention detection. It fuses data from MASS and CVSS, along with kinematic data from IMUs (worn on the forearm and upper arm) to estimate the user's intended grasp. The GPSS extracts 84 kinematic features related to displacement, velocity, and acceleration during the reaching motion. A sequential feature selection method was used to identify the most informative features. Finally, a KNN classifier (the grasp classifier) predicts the most appropriate grasp pattern for the identified object.

The overall control architecture separates into a low-level local control system (LCS) for the prosthetic hand and a high-level sensor-based control system (SCS) that fuses the multimodal inputs.

Datasets

The paper describes two phases of data collection:

  • Phase 1: Data Collection Protocol: This phase involved collecting data from 10 able-bodied participants and 1 transradial amputee. They performed pick-and-place tasks with three objects (bottle, lid, box) and seven grasp patterns across various object locations and heights. This data was used to train the object classifier (OC) and grasp classifier (GC).
  • Phase 2: Real-Time Task Conduction and Assessment: This phase evaluated the real-time performance of the trained system. Participants performed the pick-and-place tasks, and the system's ability to correctly identify objects and predict grasps was measured.

No new benchmark datasets were introduced; rather, the study utilized a controlled experimental setup to generate the necessary training and testing data.

Key Insights and Ideas

  • MMG as a Robust Alternative: The study highlights MMG as a promising alternative to EMG for prosthetic control, particularly in unstructured environments, due to its reduced sensitivity to skin conditions and motion artifacts when fused with IMU data.
  • Multimodal Fusion is Key: The success of the system hinges on the synergistic combination of different sensor modalities. Vision identifies what to grasp, MMG signals intent to act, and IMUs provide how the user is moving, collectively enabling more accurate grasp prediction.
  • Reduced Cognitive Load: By automating grasp selection based on predicted intent, the system significantly reduces the cognitive burden on the user, moving towards a more intuitive and natural interaction with the prosthesis.
  • Subject-Specific Templates are Necessary: The analysis of kinematic features revealed significant inter-subject variability, underscoring the need for subject-specific templates for accurate grasp classification.
  • Simplicity Can Be Effective: The paper demonstrates that even with relatively simple image processing techniques, valuable information can be extracted for object recognition, making the system more practical for embedded, low-power applications.

Key Results

  • High Grasp Classification Accuracy: The system achieved impressive real-time grasp classification accuracies:
  • Bottle: 100%

    • Lid: 82.5%
    • Box: 88.9% (These are combined averages across participants and object positions, excluding descending positions for able-bodied participants to match the amputee's test conditions).
  • Amputee Performance: The single amputee participant demonstrated particularly high classification accuracy, suggesting the system's potential for real-world application in prosthetic users.

  • Feature Importance: Displacement features were found to be the most discriminative kinematic features for grasp prediction, outperforming velocity and acceleration features.
  • KNN Classifier Superiority: The K-Nearest Neighbors (KNN) classifier consistently outperformed other tested classifiers (LDA, QDA, DT, SVM) for both object and grasp classification.

Limitations and Future Work

  • Limited Object Set: The study used a small set of common household objects. Future work should involve testing with a wider variety of objects with different shapes, sizes, and textures.
  • Environmental Constraints: While the system aims for practicality, the current vision processing is simplified and best suited for structured environments. Future work could explore more complex vision algorithms for more challenging conditions.
  • Single Amputee Participant: The study included only one amputee participant. An extended investigation with a larger and more diverse group of amputees is needed to validate the findings.
  • Future Work: The authors suggest several avenues for future research:
  • Expanding the object set and environmental complexity.
  • Investigating a dual EMG-MMG modality for even richer muscle signal information.
  • Developing a wearable, embedded vision system for the prosthetic device.
  • Exploring more advanced machine learning models, such as shallow neural networks.
  • Quantifying the impact on cognitive load and muscle fatigue demand.
  • Further enhancing the system for more practical and widespread commercial translation.

Summary updated

Abstract

Neurorobotic augmentation (e.g., robotic assist) is now in regular use to support individuals suffering from impaired motor functions. A major unresolved challenge, however, is the excessive cognitive load necessary for the human-machine interface (HMI). Grasp control remains one of the most challenging HMI tasks, demanding simultaneous, agile, and precise control of multiple degrees-of-freedom (DoFs) while following a specific timing pattern in the joint and human-robot task spaces. Most commercially available systems use either an indirect mode-switching configuration or a limited sequential control strategy, limiting activation to one DoF at a time. To address this challenge, we introduce a shared autonomy framework centred around a low-cost multi-modal sensor suite fusing: (a) mechanomyography (MMG) to estimate the intended muscle activation, (b) camera-based visual information for integrated autonomous object recognition, and (c) inertial measurement to enhance intention prediction based on the grasping trajectory. The complete system predicts user intent for grasp based on measured dynamical features during natural motions. A total of 84 motion features were extracted from the sensor suite, and tests were conducted on 10 able-bodied and 1 amputee participants for grasping common household objects with a robotic hand. Real-time grasp classification accuracy using visual and motion features obtained 100%, 82.5%, and 88.9% across all participants for detecting and executing grasping actions for a bottle, lid, and box, respectively. The proposed multimodal sensor suite is a novel approach for predicting different grasp strategies and automating task performance using a commercial upper-limb prosthetic device. The system also shows potential to improve the usability of modern neurorobotic systems due to the intuitive control design.
BibTeX
@inproceedings{gardner2020multimodal,
  author = {Marcus Gardner and C. Sebastian Mancero Castillo and Samuel Wilson and Dario Farina and Etienne Burdet and Boo Cheong Khoo and S. Farokh Atashzar and Ravi Vaidyanathan},
  title = {{A Multimodal Intention Detection Sensor Suite for Shared Autonomy of Upper-Limb Robotic Prostheses}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20216097},
}
Figure