Abstract
Gesture recognition for motor-impaired individuals faces significant challenges due to high signal noise, data scarcity, and the need for personalized, continual learning. We propose the Latent Embedding Exploitation (LEE) framework, which utilizes "prior knowledge" from healthy populations to guide the learning of unique gesture patterns in impaired users. Our approach employs a multi-part loss function to balance general gesture structures with individual-specific motion styles. Experimental results on real-world datasets demonstrate that LEE significantly outperforms existing few-shot learning methods, providing a robust solution for personalized wearable AI.
Introduction
Standard hand gesture recognition systems, such as those integrated into smartwatches, typically assume a "normal" range of fine motor skills. However, for individuals with conditions like Parkinson’s disease, Multiple Sclerosis, or spinal cord injuries, gesture patterns are often highly variable, noisy, and unique to the individual.
This creates three major hurdles for practical deployment:
- Data Scarcity: It is impractical to require motor-impaired individuals to perform hundreds of repetitions for model training.
- Distribution Shift: Models trained on healthy "control" participants often fail when applied to the distinct motion styles of impaired users.
- Continual Learning: Users need the ability to add custom gestures over time without the system suffering from "catastrophic forgetting" of previous tasks.
Methodology: Latent Embedding Exploitation (LEE)
The core innovation of this work is the Latent Embedding Exploitation (LEE) mechanism within a replay-based Few-Shot Continual Learning (FSCL) framework. The system navigates the feature space using three distinct latent embeddings:
- Preserved Latent Embedding ($z_c$): Derived from a feature extractor pre-trained on a large dataset of healthy control subjects, providing a baseline of general gesture structures.
- Temporary Latent Embedding ($z_i$): Captures the raw, noisy features of the current sample from the motor-impaired user.
- Learned Latent Embedding ($z_i^c$): The target embedding being optimized to represent the personalized gesture.
The framework utilizes a multi-part loss function to exploit prior knowledge (minimizing distance to $z_c$) while exploring divergence (maximizing distance from $z_i$ to capture unique user styles). To prevent forgetting, a small memory buffer—60x smaller than traditional replay buffers—stores a few examples of old gestures for periodic replay.
Datasets and Evaluation
The study utilized two primary datasets to bridge the gap between general and impaired populations:
- Source Domain: The SmartWatch Gesture Dataset, containing 3,200 sequences of 20 gestures from 8 healthy subjects.
- Target Domain: The Motion Gesture Dataset, featuring 6 motion gestures performed by 12 individuals with various upper-body motor impairments (e.g., Spinal cord injury, MS, Parkinson’s).
Key Results
The LEE method consistently outperformed established baselines such as MAML and Prototypical Networks across various disability types.
- Accuracy: On the Motion Gesture dataset, the model achieved test accuracies of 57.0% (1-shot), 64.6% (3-shot), and 69.3% (5-shot).
- Macro F1 Score: The method showed a 12.3% improvement in F1 score over competing methods, demonstrating superior handling of class imbalance and signal noise.
- Stability: Qualitative analysis via confusion matrices confirmed that LEE maintains high classification stability for previously learned gestures while adapting to new ones.
Conclusion and Future Work
The LEE framework demonstrates that "healthy" data can serve as a vital statistical anchor for personalizing AI for motor-impaired users. By explicitly modeling how a user's motion deviates from the norm, the system achieves high efficiency and accuracy on resource-constrained wearables. Future work will focus on expanding the gesture vocabulary and implementing the framework in a live, "online" wearable application to test real-time user feedback and adaptation.
Summary updated