Abstract
Most of the state-of-the-art action recognition methods focus on offline learning, where the samples of all types of actions need to be provided at once. Here, we address continual learning of action recognition, where various types of new actions are continuously learned over time. This task is quite challenging, owing to the catastrophic forgetting problem stemming from the discrepancies between the previously learned actions and current new actions to be learned. Therefore, we propose Else-Net, a novel Elastic Semantic Network with multiple learning blocks to learn diversified human actions over time. Specifically, our Else-Net is able to automatically search and update the most relevant learning blocks w.r.t. the current new action, or explore new blocks to store new knowledge, preserving the unmatched ones to retain the knowledge of previously learned actions and alleviates forgetting when learning new actions. Moreover, even though different human actions may vary to a large extent as a whole, their local body parts can still share many homogeneous features. Inspired by this, our proposed Else-Net mines the shared knowledge of the decomposed human body parts from different actions, which benefits continual learning of actions. Experiments show that the proposed approach enables effective continual action recognition and achieves promising performance on two large-scale action recognition datasets.
Introduction
In real-world applications such as home-assistant robotics or surveillance, AI models must process a continuous stream of new information. Traditional action recognition models, trained in an "offline" manner, suffer from catastrophic forgetting when learning new tasks sequentially: new gradients overwrite weights used for old tasks, causing the model to lose its ability to recognize previous actions. Else-Net addresses this by balancing plasticity (learning new skills) with stability (preserving old knowledge) through a brain-inspired modular architecture.
Proposed Method: Else-Net
Else-Net moves away from fixed-structure architectures in favor of a dynamic, modular approach composed of two primary innovations: Semantic Body Part Branches and Elastic Units.
Semantic Body Part Branches
A key insight of this work is that while a new action may be unique as a whole, its components often share homogeneous features with known actions. For instance, "throwing" and "punching" both involve similar arm movements. Else-Net exploits this by dividing the skeleton into five semantic branches: left arm, right arm, trunk, left leg, and right leg. Each branch possesses its own stack of Elastic Units, allowing the model to find the best-matched pathway for each body part independently.
Elastic Units and the Switch Block
Instead of standard layers, Else-Net utilizes "Elastic Units." Each unit contains multiple learning blocks and a Switch Block:
- Dynamic Selection: For every input, the Switch Block (using a Gumbel Softmax trick) selects the most relevant learning block.
- Expansion: If a new action is significantly different from prior knowledge, the model can "explore" by adding a new block. If useful, it is permanently integrated.
- Selective Updating: During training, only the parameters of the selected blocks are updated. The remaining blocks are frozen, acting as a stable repository for previous knowledge.
Experimental Evaluation
The researchers evaluated Else-Net on two massive benchmarks: NTU RGB+D (60 classes) and PKU-MMD (51 classes). The model was tested under a continual learning protocol where it learned 10 new categories sequentially.
Key Results
Else-Net significantly outperformed existing state-of-the-art continual learning methods like GEM and Remind:
- Accuracy (ACC): Else-Net achieved high average accuracy (e.g., 84.6% in the CS setting on NTU RGB+D), far exceeding the 71.2% of the next best method.
- Forgetting Measure (FM): Else-Net showed drastically lower forgetting (4.0%) compared to other models that lost up to 17% of prior knowledge.
- Offline Performance: Even in standard offline settings, Else-Net's modular architecture proved more powerful for feature extraction than traditional fixed-structure models.
Conclusion
Else-Net demonstrates that semantic decomposition and modular elasticity are highly effective strategies for mitigating catastrophic forgetting in action recognition. By mimicking the brain's ability to activate specific regions for specific memories, the network grows and adapts to the complexity of incoming data. Future work will focus on optimizing memory management for infinite learning scenarios and expanding the architecture to multi-modal data.
Summary updated