Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation

Riyad Bin Rafiq, Weishi Shi, Mark V. Albert

Abstract

Gesture recognition for motor-impaired individuals faces significant challenges due to high signal noise, data scarcity, and the need for personalized, continual learning. We propose the Latent Embedding Exploitation (LEE) framework, which utilizes "prior knowledge" from healthy populations to guide the learning of unique gesture patterns in impaired users. Our approach employs a multi-part loss function to balance general gesture structures with individual-specific motion styles. Experimental results on real-world datasets demonstrate that LEE significantly outperforms existing few-shot learning methods, providing a robust solution for personalized wearable AI.

Introduction

Standard hand gesture recognition systems, such as those integrated into smartwatches, typically assume a "normal" range of fine motor skills. However, for individuals with conditions like Parkinson’s disease, Multiple Sclerosis, or spinal cord injuries, gesture patterns are often highly variable, noisy, and unique to the individual.

This creates three major hurdles for practical deployment:

  1. Data Scarcity: It is impractical to require motor-impaired individuals to perform hundreds of repetitions for model training.
  2. Distribution Shift: Models trained on healthy "control" participants often fail when applied to the distinct motion styles of impaired users.
  3. Continual Learning: Users need the ability to add custom gestures over time without the system suffering from "catastrophic forgetting" of previous tasks.
Comparison of gesture signals between a control participant and a motor-impaired participant, highlighting the distribution shift and increased noise in impaired motion.

Methodology: Latent Embedding Exploitation (LEE)

The core innovation of this work is the Latent Embedding Exploitation (LEE) mechanism within a replay-based Few-Shot Continual Learning (FSCL) framework. The system navigates the feature space using three distinct latent embeddings:

  • Preserved Latent Embedding ($z_c$): Derived from a feature extractor pre-trained on a large dataset of healthy control subjects, providing a baseline of general gesture structures.
  • Temporary Latent Embedding ($z_i$): Captures the raw, noisy features of the current sample from the motor-impaired user.
  • Learned Latent Embedding ($z_i^c$): The target embedding being optimized to represent the personalized gesture.
The Latent Embedding Exploitation (LEE) framework architecture, illustrating the interaction between preserved, temporary, and learned embeddings.

The framework utilizes a multi-part loss function to exploit prior knowledge (minimizing distance to $z_c$) while exploring divergence (maximizing distance from $z_i$ to capture unique user styles). To prevent forgetting, a small memory buffer—60x smaller than traditional replay buffers—stores a few examples of old gestures for periodic replay.

Datasets and Evaluation

The study utilized two primary datasets to bridge the gap between general and impaired populations:

  • Source Domain: The SmartWatch Gesture Dataset, containing 3,200 sequences of 20 gestures from 8 healthy subjects.
  • Target Domain: The Motion Gesture Dataset, featuring 6 motion gestures performed by 12 individuals with various upper-body motor impairments (e.g., Spinal cord injury, MS, Parkinson’s).

Key Results

The LEE method consistently outperformed established baselines such as MAML and Prototypical Networks across various disability types.

  • Accuracy: On the Motion Gesture dataset, the model achieved test accuracies of 57.0% (1-shot), 64.6% (3-shot), and 69.3% (5-shot).
  • Macro F1 Score: The method showed a 12.3% improvement in F1 score over competing methods, demonstrating superior handling of class imbalance and signal noise.
  • Stability: Qualitative analysis via confusion matrices confirmed that LEE maintains high classification stability for previously learned gestures while adapting to new ones.

Conclusion and Future Work

The LEE framework demonstrates that "healthy" data can serve as a vital statistical anchor for personalizing AI for motor-impaired users. By explicitly modeling how a user's motion deviates from the norm, the system achieves high efficiency and accuracy on resource-constrained wearables. Future work will focus on expanding the gesture vocabulary and implementing the framework in a live, "online" wearable application to test real-time user feedback and adaptation.

Summary updated

Abstract

Hand gestures can provide a natural means of human-computer interaction and enable people who cannot speak to communicate efficiently. Existing hand gesture recognition methods heavily depend on pre-defined gestures, however, motor-impaired individuals require new gestures tailored to each individual's gesture motion and style. Gesture samples collected from different persons have distribution shifts due to their health conditions, the severity of the disability, motion patterns of the arms, etc. In this paper, we introduce the Latent Embedding Exploitation (LEE) mechanism in our replay-based Few-Shot Continual Learning (FSCL) framework that significantly improves the performance of fine-tuning a model for out-of-distribution data. Our method produces a diversified latent feature space by leveraging a preserved latent embedding known as gesture prior knowledge, along with intra-gesture divergence derived from two additional embeddings. Thus, the model can capture latent statistical structure in highly variable gestures with limited samples. We conduct an experimental evaluation using the SmartWatch Gesture and the Motion Gesture datasets. The proposed method results in an average test accuracy of 57.0%, 64.6%, and 69.3% by using one, three, and five samples for six different gestures. Our method helps motor-impaired persons leverage wearable devices, and their unique styles of movement can be learned and applied in human-computer interaction and social communication. Code is available at: https://github.com/riyadRafiq/wearable-latent-embedding-exploitation
BibTeX
@inproceedings{rafiq2024wearable,
  author = {Riyad Bin Rafiq and Weishi Shi and Mark V. Albert},
  title = {{Wearable Sensor-Based Few-Shot Continual Learning on Hand Gestures for Motor-Impaired Individuals via Latent Embedding Exploitation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure