Data-Free Class-Incremental Gesture Recognition with Prototype-Guided Pseudo Feature Replay

Hongsong Wang, Ao Sun, Jie Gui, Liang Wang

Abstract

Gesture recognition is an important research area in the field of computer vision. Most gesture recognition efforts focus on close-set scenarios, thereby limiting the capacity to effectively handle unseen or novel gestures. We aim to address class-incremental gesture recognition, which entails the ability to accommodate new and previously unseen gestures over time. Specifically, we introduce a Prototype-Guided Pseudo Feature Replay (PGPFR) framework for data-free class-incremental gesture recognition. This framework comprises four components: Pseudo Feature Generation with Batch Prototypes (PFGBP), Variational Prototype Replay (VPR) for old classes, Truncated Cross-Entropy (TCE) for new classes, and Continual Classifier Re-Training (CCRT). To tackle the issue of catastrophic forgetting, the PFGBP dynamically generates a diversity of pseudo features in an online manner, leveraging class prototypes of old classes along with batch class prototypes of new classes. Furthermore, the VPR enforces consistency between the classifier's weights and the prototypes of old classes, leveraging class prototypes and covariance matrices to enhance robustness and generalization capabilities. The TCE mitigates the impact of domain differences of the classifier caused by pseudo features. Finally, the CCRT training strategy is designed to prevent overfitting to new classes and ensure the stability of features extracted from old classes. Extensive experiments conducted on two widely used gesture recognition datasets, namely SHREC 2017 3D and EgoGesture 3D, demonstrate that our approach outperforms existing state-of-the-art methods by 11.8% and 12.8% in terms of mean global accuracy, respectively.

Introduction

In the field of AI, "catastrophic forgetting" remains a significant challenge. When a model trained on a specific set of gestures is introduced to new ones, it often overwrites its previous knowledge, losing the ability to identify original classes. While standard solutions involve "replay"—storing a subset of old data to retrain the model—this is often impossible in sensitive applications like healthcare or smart-home control due to privacy regulations (e.g., GDPR) or hardware memory constraints.

Conceptual overview of the PGPFR framework for data-free class-incremental gesture recognition.

This paper addresses Data-Free Class-Incremental Learning (CIL), proposing a framework that allows models to learn new hand signs over time without storing any original user data. The proposed PGPFR (Prototype-Guided Pseudo Feature Replay) framework leverages class "prototypes" and covariance matrices to generate synthetic features for rehearsal, maintaining high performance across both old and new tasks.

Methodology

The PGPFR framework is built upon four primary technical pillars designed to balance plasticity (learning new things) and stability (remembering old things):

  1. Pseudo Feature Generation with Batch Prototypes (PFGBP): Instead of storing raw data, the model stores mathematical averages (prototypes) of features. During training on new tasks, it uses these prototypes and geometric transformations to "hallucinate" pseudo-features of old gestures, which are then mixed with new data.
  2. Variational Prototype Replay (VPR): To increase the realism of hallucinated features, the model stores covariance matrices. This allows it to treat old classes as distributions (clouds of possibilities) rather than single points, enhancing robustness against variations in gesture performance.
  3. Truncated Cross-Entropy (TCE): This specialized loss function addresses the "domain gap" between real features and synthetic ones, preventing the model from becoming confused by the inherent differences in synthetic data quality.
  4. Continual Classifier Re-Training (CCRT): To prevent knowledge drift, the feature extractor (backbone) is fixed after the initial task. Subsequent training focuses on the final classification layers to ensure the stability of features extracted from old classes.
Detailed architecture of the PGPFR framework, illustrating the interaction between tasks through pseudo-feature generation and variational replay.

Experimental Evaluation

The model was evaluated on two major 3D gesture recognition datasets: SHREC 2017 3D (14 gesture types, 22 hand joints) and EgoGesture 3D (83 types of static and dynamic gestures).

Key Results

  • Accuracy: PGPFR significantly outperformed the previous state-of-the-art (BOAT-MI), showing a 12.8% improvement in mean global accuracy on SHREC 2017 3D and an 11.8% improvement on EgoGesture 3D.
  • Efficiency: Unlike previous data-free methods that rely on slow "model inversion" to recreate images, PGPFR operates directly in the feature space. This makes it approximately 250 times faster to train (1,680 seconds vs. 415,260 seconds for competing methods).
  • Model Size: With only 0.27M parameters, the framework is highly suitable for deployment on edge devices like wearables or mobile phones.

Conclusion and Limitations

PGPFR demonstrates that high-performance incremental learning is possible without compromising user privacy. By using batch prototypes and covariance-based replay, the model achieves a superior balance between learning and retention.

However, limitations remain. The current pseudo-feature generation relies on linear operations, which may struggle to replicate the complex, non-linear distributions of real-world human movement. Specifically, the model can still experience confusion when a new gesture is highly similar to an old one (e.g., different directions of the same swipe). Future research will focus on more sophisticated modeling of these overlapping class boundaries while maintaining the efficiency and privacy benefits of the data-free approach.

Summary updated

Abstract

Gesture recognition is an important research area in the field of computer vision. Most gesture recognition efforts focus on close-set scenarios, thereby limiting the capacity to effectively handle unseen or novel gestures. We aim to address class-incremental gesture recognition, which entails the ability to accommodate new and previously unseen gestures over time. Specifically, we introduce a Prototype-Guided Pseudo Feature Replay (PGPFR) framework for data-free class-incremental gesture recognition. This framework comprises four components: Pseudo Feature Generation with Batch Prototypes (PFGBP), Variational Prototype Replay (VPR) for old classes, Truncated Cross-Entropy (TCE) for new classes, and Continual Classifier Re-Training (CCRT). To tackle the issue of catastrophic forgetting, the PFGBP dynamically generates a diversity of pseudo features in an online manner, leveraging class prototypes of old classes along with batch class prototypes of new classes. Furthermore, the VPR enforces consistency between the classifier's weights and the prototypes of old classes, leveraging class prototypes and covariance matrices to enhance robustness and generalization capabilities. The TCE mitigates the impact of domain differences of the classifier caused by pseudo features. Finally, the CCRT training strategy is designed to prevent overfitting to new classes and ensure the stability of features extracted from old classes. Extensive experiments conducted on two widely used gesture recognition datasets, namely SHREC 2017 3D and EgoGesture 3D, demonstrate that our approach outperforms existing state-of-the-art methods by 11.8\% and 12.8\% in terms of mean global accuracy, respectively. The code is available on https://github.com/sunao-101/PGPFR-3/.
BibTeX
@misc{wang2025data,
  author = {Hongsong Wang and Ao Sun and Jie Gui and Liang Wang},
  title = {{Data-Free Class-Incremental Gesture Recognition with Prototype-Guided Pseudo Feature Replay}},
  year = {2025},
  eprint = {2505.20049},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.20049},
}
Figure