Combining Electromyography and Tactile Myography to Improve Hand and Wrist Activity Detection in Prostheses

Noémie Jaquier, Mathilde Connan, Claudio Castellini, Sylvain CalinonTechnologies 2017 · DOI

TL;DR

This paper explores using a novel tactile myography (TMG) sensor, which measures muscle bulging, to improve control of prosthetic hands. By combining TMG with traditional electromyography (sEMG) and employing Gaussian Process Regression (GPR), the researchers achieved more accurate and proportional detection of hand and wrist movements, a critical step towards more intuitive prosthetic control.

What Problem Does This Paper Address?

Controlling advanced prosthetic hands with multiple degrees of freedom (DOFs) proportionally and simultaneously remains a significant challenge. Current methods primarily rely on surface electromyography (sEMG) to capture muscle electrical activity. However, sEMG signals can be noisy and limited in resolution, making it difficult to accurately infer user intent, especially for complex movements. This often leads to a limited number of controllable DOFs and a less natural user experience. The paper aims to address this by investigating a new sensor modality, tactile myography (TMG), which captures muscle deformation, and by exploring advanced machine learning techniques to fuse information from both sEMG and TMG.

Methods

The core of the paper lies in applying Gaussian Process Regression (GPR) for activity detection. GPR is chosen for its ability to learn from limited data, which is crucial for practical prosthetic applications where extensive training data collection can be burdensome for users.

Key methodological contributions include:

  • Tactile Myography (TMG): The paper utilizes a novel, high-resolution tactile bracelet composed of 320 sensitive cells arranged in an array. This sensor captures the physical bulging of forearm muscles, providing a different modality of muscle activity information compared to sEMG.
  • Multimodal Regression: The researchers propose a GPR framework that can effectively combine data from multiple sensor modalities (sEMG and TMG). This is achieved by defining a kernel function that is a product of individual kernels for each modality.
  • Kernel Function Design: For GPR, various kernel distances were investigated, including Euclidean, SPD (Symmetric Positive Definite), logSPD, and Kullback-Leibler (KL) divergence, to find the most effective way to measure similarity between sensor data points.
  • Real-time vs. Batch Processing: The study compares performance in both a real-time goal-reaching experiment and a batch processing scenario, providing insights into practical usability and offline performance.

Datasets

The study used data collected from nine non-amputated right-handed participants.

  • Training Data: For the real-time experiment, training data consisted of recordings of specific movements (wrist flexion/extension, wrist supination, thumb/index/little-finger flexion) performed at zero and full activation levels. For the batch experiment, data from zero and full activation were used as training sets.
  • Testing Data: In the real-time experiment, testing involved controlling a virtual hand in a goal-reaching task with varying activation levels. In the batch experiment, testing data comprised recordings from the transition phases of movements, representing intermediate activation levels.
  • Ground Truth: The "ground truth" for hand and wrist movements was derived from the synchronized visual stimulus of an animated 9-DOF hand model that participants were asked to imitate. This approach was chosen to simulate a scenario where direct hand motion capture is not feasible, as would be the case for amputees.

Key Insights and Ideas

  • TMG's Promise: The tactile myography sensor shows significant potential, often outperforming traditional sEMG in detecting hand and wrist movements, especially when used with GPR. This suggests that capturing the physical deformation of muscles offers valuable information for prosthetic control.
  • GPR's Advantage: Gaussian Process Regression consistently outperformed a simpler linear regression method (Ridge Regression) in terms of success rate and accuracy (RMSE), particularly in the real-time experiment. This highlights the benefit of a non-linear, probabilistic approach for complex muscle signal interpretation.
  • Multimodal Fusion Benefits: While TMG alone showed strong performance, combining TMG with sEMG showed potential for further improvement, especially with specific kernel configurations. This indicates that leveraging complementary sensor information can lead to more robust control.
  • Kernel Choice Matters: The choice of distance measure within the GPR kernel significantly impacts performance. For TMG, Euclidean and logSPD distances showed good results, while for multimodal inputs, Euclidean/Euclidean and SPD/SPD combinations were effective.
  • Real-time vs. Batch Trade-offs: The real-time experiment, while more indicative of practical usability, showed slightly lower success rates compared to the batch experiment. This is likely due to the inherent challenges of online adaptation and the dynamic nature of real-time control.

Key Results

  • Experiment 1 (Real-time TMG): GPR achieved a significantly higher success rate (55.35%) compared to Ridge Regression (35.60%) for detecting hand and wrist movements using TMG data. GPR also reduced the time to complete tasks and had a lower RMSE.
  • Experiment 2 (Batch Comparison):
    • TMG generally outperformed sEMG for both RR and GPR when used as unimodal inputs.
  • GPR consistently outperformed RR across various sensor inputs and distance measures.
  • Combining sEMG and TMG showed improvements over unimodal inputs for some configurations, particularly with RR. For GPR, combining sensors with specific distance measures (e.g., Euclidean/Euclidean) also yielded benefits.
  • The RMSE for TMG was generally lower than for sEMG, indicating better prediction accuracy.
  • Wrist supination was identified as the easiest movement to detect, while index flexion and wrist extension were the most challenging.

Limitations and Future Work

  • Ground Truth Proxy: The use of an animated hand model as ground truth, while necessary for non-amputee studies, introduces a potential mismatch with actual user intent and may reduce prediction precision.
  • Offline vs. Online Performance: The paper acknowledges that offline accuracy metrics may not fully reflect practical usability in real-world scenarios.
  • Participant Adaptation: The study notes that the impact of participant adaptation to the visual stimuli on prediction accuracy is not fully investigated.
  • Kernel Exploration: While RBF kernels were used due to their effectiveness and ease of tuning, future work could explore other kernel types (e.g., Matérn kernels).
  • Real-time Evaluation: Further real-time evaluations are planned to assess the proposed techniques in more dynamic and realistic daily living setups.
  • Multimodal Potential: The authors plan to further investigate the potential of multimodal inputs and compare different kernel configurations in real-time goal-reaching experiments.

Summary updated

Abstract

Despite recent advances in prosthetics and assistive robotics in general, robust simultaneous and proportional control of dexterous prosthetic devices remains an unsolved problem, mainly because of inadequate sensorization. In this paper, we study the application of regression to muscle activity, detected using a flexible tactile sensor recording muscle bulging in the forearm (tactile myography—TMG). The sensor is made of 320 highly sensitive cells organized in an array forming a bracelet. We propose the use of Gaussian process regression to improve the prediction of wrist, hand and single-finger activation, using TMG, surface electromyography (sEMG; the traditional approach in the field), and a combination of the two. We prove the effectiveness of the approach for different levels of activations in a real-time goal-reaching experiment using tactile data. Furthermore, we performed a batch comparison between the different forms of sensorization, using a Gaussian process with different kernel distances.
BibTeX
@inproceedings{jaquier2017combining,
  author = {Noémie Jaquier and Mathilde Connan and Claudio Castellini and Sylvain Calinon},
  title = {{Combining Electromyography and Tactile Myography to Improve Hand and Wrist Activity Detection in Prostheses}},
  booktitle = {Technologies},
  year = {2017},
  doi = {10.3390/technologies5040064},
}
Figure