Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation

Riyad Bin Rafiq, Weishi Shi, Mark V. Albert

Abstract

Hand gestures provide a natural means of human-computer interaction (HCI) and enable individuals with speech impairments to communicate efficiently. However, existing gesture recognition methods often rely on pre-defined, standardized movements that do not account for the unique motion patterns of motor-impaired individuals. These users require personalized gestures tailored to their specific physical conditions, which often result in significant distribution shifts from healthy control data. This paper introduces the Latent Embedding Exploitation (LEE) mechanism within a replay-based Few-Shot Continual Learning (FSCL) framework. By leveraging preserved latent embeddings as gesture prior knowledge and encouraging intra-gesture divergence, our method captures the statistical structure of highly variable gestures using very few samples. Experimental results on the SmartWatch Gesture and Motion Gesture datasets demonstrate average test accuracies of 57.0%, 64.6%, and 69.3% using one, three, and five samples, respectively, for six different gestures. This approach facilitates the use of wearable devices for HCI and social communication among motor-impaired populations.

Introduction

Most modern gesture recognition systems are built on the assumption that users can perform crisp, consistent, and standardized movements. For individuals with motor impairments—such as those living with Parkinson’s disease, Multiple Sclerosis (MS), or spinal cord injuries—this assumption is rarely met. Their movements are often characterized by high variability, noise, and individualized styles.

Furthermore, collecting the thousands of data samples typically required to train deep learning models is physically taxing for these users. This necessitates a system capable of Few-Shot Learning (FSL)—learning from minimal examples—and Continual Learning (CL)—adding new gestures over time without forgetting previously learned ones. Standard transfer learning often fails in these scenarios because the "distribution shift" between healthy subjects and motor-impaired individuals is too vast. To address this, we propose the Latent Embedding Exploitation (LEE) framework.

Methods: Latent Embedding Exploitation (LEE)

The LEE mechanism is integrated into a replay-based continual learning framework. Rather than relying on a single feature representation, the model utilizes three distinct latent embeddings to navigate the complex feature space of impaired motor signals:

  1. Preserved Latent Embedding ($z_c$): This serves as "gesture prior knowledge." Derived from a model pre-trained on healthy control subjects, this embedding remains fixed to provide a stable baseline of fundamental hand movement physics.
  2. Temporary Sample Embedding ($z_i$): A snapshot of the current feature extraction for a new, incoming sample.
  3. Learned Latent Embedding ($z_i^c$): The active embedding that is updated during training to adapt to the specific user's unique motion style.

Optimization and Divergence

The core innovation of LEE lies in its loss functions. The model balances classification accuracy with two structural objectives:

  • Similarity to the Prior: Ensuring that learned features do not drift too far from the basic structural reality of hand gestures.
  • Intra-gesture Divergence: Unlike traditional models that seek to minimize variance, LEE explicitly encourages the model to explore a wider latent space. This allows the system to capture the inherent noise and variability in impaired motor signals, making the model more robust to gestures that are never performed exactly the same way twice.

To prevent "catastrophic forgetting," the framework employs a small replay buffer of past samples. Notably, this buffer is approximately 60x smaller than those used in traditional replay-based methods, making it suitable for resource-constrained wearable devices.

Experimental Evaluation

Datasets

The study utilized two primary datasets to evaluate the bridge between general and specialized motion:

  • Source Domain (SmartWatch Gesture Dataset): 3,200 sequences of 20 gestures performed by eight healthy subjects using a Sony smartwatch.
  • Target Domain (Motion Gesture Dataset): Gestures from 12 participants with various upper-body motor impairments (Spinal cord injury, Parkinson’s, MS, etc.), collected via a Samsung Gear Fit 2.

Performance Metrics

The LEE framework was compared against several baselines, including Model-Agnostic Meta-Learning (MAML), iCaRL, and standard LSTM fine-tuning.

  • Accuracy: For users with spinal cord injuries, LEE achieved 57.0% accuracy with 1-shot learning, rising to 69.3% with 5-shot learning across six gestures.
  • Stability: While baseline methods like iCaRL showed significant performance degradation as the number of gestures increased, LEE maintained a stable performance curve, demonstrating superior resistance to catastrophic forgetting.
  • F1-Score: LEE showed a 12.3% improvement in macro F1-score over the next best method, indicating a superior ability to distinguish between similar, noisy gesture patterns.

Conclusion and Future Work

The LEE framework demonstrates that "normal" data from healthy subjects should not be discarded when training for impaired users; instead, it should serve as a latent anchor. By embracing intra-gesture divergence, the model successfully adapts to the high-variance signals characteristic of motor impairments.

Future research will focus on mitigating the impact of gesture learning order and transitioning the framework into a real-time, online learning application. This would allow wearable devices to continuously improve recognition during daily use, further enhancing the independence and communication capabilities of motor-impaired individuals.

Summary updated

Abstract

Hand gestures can provide a natural means of human-computer interaction and enable people who cannot speak to communicate efficiently. Existing hand gesture recognition methods heavily depend on pre-defined gestures, however, motor-impaired individuals require new gestures tailored to each individual's gesture motion and style. Gesture samples collected from different persons have distribution shifts due to their health conditions, the severity of the disability, motion patterns of the arms, etc. In this paper, we introduce the Latent Embedding Exploitation (LEE) mechanism in our replay-based Few-Shot Continual Learning (FSCL) framework that significantly improves the performance of fine-tuning a model for out-of-distribution data. Our method produces a diversified latent feature space by leveraging a preserved latent embedding known as gesture prior knowledge, along with intra-gesture divergence derived from two additional embeddings. Thus, the model can capture latent statistical structure in highly variable gestures with limited samples. We conduct an experimental evaluation using the SmartWatch Gesture and the Motion Gesture datasets. The proposed method results in an average test accuracy of 57.0%, 64.6%, and 69.3% by using one, three, and five samples for six different gestures. Our method helps motor-impaired persons leverage wearable devices, and their unique styles of movement can be learned and applied in human-computer interaction and social communication. Code is available at: https://github.com/riyadRafiq/wearable-latent-embedding-exploitation
BibTeX
@inproceedings{rafiq2024wearable,
  author = {Riyad Bin Rafiq and Weishi Shi and Mark V. Albert},
  title = {{Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure